Development of an Evaluation Framework for Health Information Systems (DIPSA)

Health information communication technology evaluation frameworks for pharmacist prescribing: A systematic scoping review

BACKGROUND

Information communication technology (ICT) is instrumental in pharmacists' current practice and emerging roles. One such role is prescribing, which requires the use of clinical guidelines and documentation of decision-making, commonly via ICT. Development and refinement of ICT should be guided by evaluation frameworks that describe or measure features of ICT and its implementation. In the context of pharmacist prescribing, these evaluation frameworks should be specific to health stakeholders and the pharmacy setting.

OBJECTIVES

To identify ICT evaluation frameworks from health-related literature and review frameworks relevant to the development, implementation, and evaluation of pharmacist prescribing.

METHODS

A database search of CINAHL, Cochrane Library, EMBASE, Medline (Ovid), ProQuest, Scopus, Web of Science and grey literature was conducted, using combinations of keywords relating to 'ICT', 'utilization', 'usability', and 'evaluation framework'. Abstracts and titles were screened according to inclusion criteria. Identified evaluation frameworks were critiqued for relevance to pharmacy practice.

RESULTS

Twenty-two articles were identified, describing the development or application of 20 evaluation frameworks. None of the frameworks was developed specifically for pharmacy practice. The Technology Acceptance Model (TAM), describing use behavior, behavior intention, perceived usefulness, and perceived ease of use, was the most widely utilized framework. The Information System Success (ISS) and Human-Organization and Technology Fit (HOT-fit) are notable evaluation frameworks that address user and organizational influences in health ICT utility, and factors of both can address the limitation of TAM.

CONCLUSIONS

The findings call for development of an agile evaluation framework for the system under review; however, this can prove difficult due to the heterogenicity and complexity of the healthcare system, particularly contemporary pharmacy practice. While the TAM appears useful to evaluate user attitudes and intentions towards ICT, its relevance to ICT in contemporary community pharmacy practice requires exploration.

HIET Web-based digital repository for health informatics evaluation tools

Purpose

Evaluating health information systems is an integral part of the life cycle and development of information systems as it can improve the quality of health care. The purpose of this paper is to introduce a bilingual Web-based repository of health-related software products evaluation tools.

Design/methodology/approach

The present paper is an applied-developmental study that includes the stages of analysis, design, implementation and evaluation procedures. By searching valid databases as well as holding focus group meetings with a group of experts, the necessary elements for designing a Web-based repository were identified, and also unified modelling language diagrams were designed by using a visual paradigm. The coding(programming) was conducted based on the Gantlet Web Systems Development Framework at the next stage. Finally, after implementing and testing the system, the content was added to the repository, and then the repository was evaluated in terms of usability testing.

Findings

The health informatics evaluation tools (HIET) repository provides a functional and selective environment that facilitates the sharing, online storage and retrieval of assessment tools by the scientific community. The HIET repository is easily accessible at www.hiet.ir/ The website is implemented in structured query language (MySQL), personal homepage. Hypertext Preprocessor (PHP) and Linux, Apache, MySQL, and PHP (LEMP) and supports all major browsers.

Originality/value

The HIET repository, as mentioned earlier, serves as an application environment for sharing, storing and online retrieving the assessment tools of health information systems. Therefore, this tool not only facilitates the search, retrieving and study of many evaluation-related papers, which are time-consuming and stressful for researchers and students but can lead to a faster and more scientific evaluation of information systems.

Electronic medical record implementation in a large healthcare system from a leadership perspective

Background

Information on the use of change management models to guide electronic medical records (EMR) implementation is limited. This case study describes the leadership aspects of a large-scale EMR implementation using Kotter’s change management model.

Methods

This case study presents the experience in implementing a new EMR system from the leadership perspective at King Abdulaziz Medical City, a large tertiary care hospital in Riyadh, Kingdom of Saudi Arabia. We described the process of implementation and outlined the challenges and opportunities, throughout the journey from the pre-implementation to the post-implementation phases.

Results

We described the corresponding actions to the eight domains of Kotter’s change management model: creating a sense of urgency, building the guiding team, developing a change vision and strategy, understanding and buy-in, removing obstacles, creating short-term wins, building on the change and anchoring the changes in corporate culture.

Conclusions

The case study highlights that EMR implementation is not a pure information technology project but rather is a technical-based complex social adaptive project that requires a specific set of leadership competencies that are central to its success. It demonstrates that change management models might be useful for large-scale EMR implementation.

A multivariate statistical evaluation of actual use of electronic health record systems implementations in Kenya

Background

Health facilities in developing countries are increasingly adopting Electronic Health Records systems (EHRs) to support healthcare processes. However, only limited studies are available that assess the actual use of the EHRs once adopted in these settings. We assessed the state of the 376 KenyaEMR system (national EHRs) implementations in healthcare facilities offering HIV services in Kenya.

Methods

The study focused on seven EHRs use indicators. Six of the seven indicators were programmed and packaged into a query script for execution within each KenyaEMR system (KeEMRs) implementation to collect monthly server-log data for each indicator for the period 2012–2019. The indicators included: Staff system use, observations (clinical data volume), data exchange, standardized terminologies, patient identification, and automatic reports. The seventh indicator (EHR variable Completeness) was derived from routine data quality report within the EHRs. Data were analysed using descriptive statistics, and multiple linear regression analysis was used to examine how individual facility characteristics affected the use of the system.

Results

213 facilities spanning 19 counties participated in the study. The mean number of authorized users who actively used the KeEMRs was 18.1% (SD = 13.1%, p<0.001) across the facilities. On average, the volume of clinical data (observations) captured in the EHRs was 3363 (SD = 4259). Only a few facilities(14.1%) had health data exchange capability. 97.6% of EHRs concept dictionary terms mapped to standardized terminologies such as CIEL. Within the facility EHRs, only 50.5% (SD = 35.4%, p< 0.001) of patients had the nationally-endorsed patient identifier number recorded. Multiple regression analysis indicated the need for improvement on the mode of EHRs use of implementation.

Conclusion

The standard EHRs use indicators can effectively measure EHRs use and consequently determine success of the EHRs implementations. The results suggest that most of the EHRs use areas assessed need improvement, especially in relation to active usage of the system and data exchange readiness.

Evaluation frameworks for digital nursing technologies: analysis, assessment, and guidance. An overview of the literature

Background

The evaluation of digital nursing technologies (DNT) plays a major role in gaining knowledge about certain aspects of a technology such as acceptance, effectiveness, or efficiency. Evaluation frameworks can help to classify the success or failure of a DNT or to further develop the technology. In general, there are many different evaluation frameworks in the literature that provide overviews of a wide variety of aspects, which makes this a highly diverse field and raises the question how to select a suitable framework. The aim of this article is to provide orientation in the field of comprehensive evaluation frameworks that can be applied to the field of DNT and to conduct a detailed analysis and assessment of these frameworks to guide field researchers.

Methods

This overview was conducted using a three-component search process to identify relevant frameworks. These components were (1) a systematized literature search in PubMed; (2) a narrative review and (3) expert consultations. Data relating to the frameworks’ evaluation areas, purpose, perspectives, and success definitions were extracted. Quality criteria were developed in an expert workshop and a strength and weakness assessment was carried out.

Results

Eighteen relevant comprehensive evaluation frameworks for DNT were identified. Nine overarching evaluation areas, seven categories of purposes, five evaluation perspectives and three categories of success definitions could be identified. Eleven quality criteria for the strengths and weaknesses of DNT-related evaluation frameworks were developed and the included frameworks were assessed against them.

Conclusion

Evaluators can use the concise information and quality criteria of this article as a starting point to select and apply appropriate DNT evaluation frameworks for their research projects or to assess the quality of an evaluation framework for DNT, as well as a basis for exploring the questions raised in this article. Future research could address gaps and weaknesses in existing evaluation frameworks, which could improve the quality of future DNT evaluations.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12912-021-00654-8.

Apply the SERVQUAL Instrument to Measure Service Quality for the Adaptation of ICT Technologies: A Case Study of Nursing Homes in Taiwan

The adoption of information and communication technology by elderly care organizations is an inevitable trend. Most empirical studies on e-Health service quality have focused predominantly on the general population rather than on the elderly. Thus, the generalizations are rather problematic. In addition, in the planning stage, pre-implementation analysis is considered critical but seldom performed. In this research, an instrument to evaluate the e-Health service quality in nursing homes was developed based on the SERVQUAL model. Furthermore, a pre-implementation analysis combining the SERVQUAL questionnaire and importance performance analysis was performed. Dissatisfactory factors were identified as follows. Regarding the physical environment quality, the residents expressed that the nursing homes did not provide well-maintained rooms and that the temperature in the rooms was unsuitable. Regarding the outcome quality, the elderly residents replied that the medical treatments and doctor visits were not well scheduled. Regarding the interaction quality, the residents indicated that the staff did not solve their problems sincerely or clearly understand their needs. Health care informatics (HCI) such as an electronic shift system (ESS) and electronic health records (EHR) are proposed to eliminate these problems. Given current resource limitations, our instrument and methodology proposed in this research could be extremely meaningful in practical application.

Evaluation of Health Information System (HIS) in The Surveillance of Dengue in Indonesia: Lessons from Case in Bandung, West Java

This study was performed to evaluate the health information system regarding the dengue surveillance system in Indonesia. Major obstacles to the implementation of an effective health information system regarding dengue cases in Bandung are examined, and practical suggestions on measures to overcome them are discussed. The study utilized a mixed-method research design using qualitative approaches: document analysis, key informants and focus group interviews. Thirty key informants were selected, comprised of policymakers, senior managers, and staff at the Ministry of Health. Data from documents and transcripts were evaluated through a modified Institutional Analysis and Development (IAD) framework described by Ostrom. Through this study, we have identified several issues that hinder the effective implementation of the health information system in the case of dengue in Bandung. In the end, we propose several recommendations for reform that encompasses motivational, strategic, and structural approaches to each component of the analysis. Through evaluation of the health information system for dengue surveillance in Indonesia, we conclude that well-coordination in multi-level governance in a country as large as Indonesia is the key in the implementation of the health information system in different levels of agencies. Furthermore, the adaptability of human resources in adopting a new information system also plays an important part.

Evaluation of the Integrated Health Information System (IHIS) in Public Hospitals in Cyprus Utilizing the DIPSA Framework

INTRODUCTION

The Ministry of Health of the Republic of Cyprus has implemented an Integrated Health Information System (IHIS) in two hospitals. However, no evaluation of IHIS has been conducted to assess its safety, efficiency and effectiveness. The proper utilization of IHIS is essential for the provision of quality healthcare services.

AIM

The purpose of this study was to evaluate the current IHIS in public hospitals in Cyprus utilizing the DIPSA evaluation framework.

METHODS

A total of 309 subjects, including doctors, nurses and other healthcare professionals, participated in the study. The DIPSA evaluation framework assessed the users' perception in five categories namely, satisfaction, collaboration, system quality, safety and procedures, using Likert scale and 3 open questions. Correlation between the categories was assessed using the Pearson correlation coefficient, and multiple regression analysis was used to examine the relationship between the demographic characteristics and categories. Data analysis was done using SPSS v24.

RESULTS

All five categories were rated moderately, between 2.5 and 3, by the participants. All categories were correlated (P < 0.01). Multiple regression analysis indicated the need for improvement between the professionals (mainly doctors and nurses) and the categories. The open questions pointed out the need for improvement in all 3 factors examined (Technology, Human Factor, Organization).

DISCUSSION

The moderately rated categories, in the Cyprus IHIS, suggest that there is a lot of room for improvement. Some interventions are suggested that could positively and simultaneously affect one or more categories.