Health information systems evaluation frameworks: A systematic review

The Sri Lankan enigma: demystifying public healthcare information systems acceptance

The deployment of Health Information Systems (HIS) in Sri Lanka has been low in adoption compared to developed countries. There has been a dearth of studies to identify the factors that improve the adoption of HIS in developing countries. Thus, this study investigates the factors influencing the acceptance of HIS among public healthcare staff. A survey was administered among 170 medical professionals, including nurses and doctors. Partial Least Squares Structural Equation Modelling (PLS-SEM) was applied to the dataset with 5000 bootstrap subsamples. The research model was developed based on the prior literature and by extending the Technology Acceptance Model (TAM) to the context of public healthcare. A positive relationship was observed between the actual use of HIS and constructs such as perceived usefulness, perceived ease of use, attitude, behavioural intention, prior use of HIS by supervisors, computer anxiety and facilitating conditions. These findings confirm the applicability of the proposed extended TAM in the public healthcare system of a developing country. Furthermore, HIS practitioners and policymakers in the healthcare sector would find these results valuable.

A process for contextualising digital health terminology standards for Uganda's health information systems: A use case of HIV information management services

Background: Uniform interpretation of digital health messages is important to achieve semantic interoperability of electronic health information systems (eHIS). Whereas international digital health terminologies such as ICD, LOINC and SNOMED-CT exist, their design considerations regarding health processes, data collected, and technologies, among others do not necessarily match Uganda's eHIS contextual needs. Objective: This research aimed to design a process that could be used to contextualise international digital health terminologies for Uganda's eHIS. Methods: The Design Science approach was used in designing the contextualisation process while utilising a foundation contextualisation approach for mapping terminologies. Results: The contextualisation process constitutes six major phases; assessing the national digital health information system context, extracting data elements in the national digital health information system, mapping existing national data elements to international terminologies, identifying and coding unmatched data elements, validating contextualised terminologies and digitising the validated terminologies. The terminology standards contextualisation process was validated using the Delphi technique and the HIV Information Management Services use case. The validation results showed that the contextualisation process was relevant, usable, adaptable and interoperable to Uganda's eHIS. Conclusion: Accordingly, this study demonstrated how international digital health terminologies could be contextualised for Uganda's health information systems. The contextualisation process could also be applied to other disease information management services in Uganda.

Electronic health record and primary care physician self-reported quality of care: a multilevel study in China

BACKGROUND

Health information technology is one of the building blocks of a high-performing health system. However, the evidence regarding the influence of an electronic health record (EHR) on the quality of care remains mixed, especially in low- and middle-income countries.

OBJECTIVE

This study examines the association between greater EHR functionality and primary care physician self-reported quality of care.

METHODS

A total of 224 primary care physicians from 38 community health centres (CHCs) in four large Chinese cities participated in a cross-sectional survey to assess CHC care quality. Each CHC director scored their CHC's EHR functionality on the availability of ten typical features covering health information, data, results management, patient access, and clinical decision support. Data analysis utilised hierarchical linear modelling.

RESULTS

The availability of five EHR features was positively associated with physician self-reported clinical quality: share records online with providers outside the practice (β = 0.276, p = 0.04), access records online by the patient (β = 0.325, p = 0.04), alert provider of potential prescription problems (β = 0.353, p = 0.04), send the patient reminders for care (β = 0.419, p = 0.003), and list patients by diagnosis or health risk (β = 0.282, p = 0.04). However, no association was found between specific features availability or total features score and physician self-reported preventive quality.

CONCLUSIONS

This study provides evidence that the availability of EHR systems, and specific features of these systems, was positively associated with physician self-reported quality of care in these 38 CHCs. Future longitudinal studies focused on standardised quality metrics, and designed to control known confounding variables, will further inform quality improvement efforts in primary care.

Exploring the feasibility of a culturally tailored infant nutrition intervention: a qualitative study of perspectives from community facilitators and attendees in a pilot randomised controlled trial - Nurture Early for Optimal Nutrition (NEON) in East London

Introduction

Appropriate and healthy feeding practices can enhance a child's health, prevent obesity and reduce chronic metabolic disease risks. Given the ethnic variations in feeding practices and metabolic risk, interventions must be community specific. Culturally tailored grassroots interventions targeting infant feeding can induce behavioural changes, mitigating chronic metabolic disease risks in later life.The aim of this study was to explore participant feedback and inform intervention delivery methods within marginalised communities.

Methods

A pilot three-arm cluster randomised controlled trial was conducted in the Tower Hamlets and Newham boroughs of London, involving community participatory learning and action groups. The study recruited 186 South Asian (Indian, Bangladeshi, Pakistani and Sri Lankan) mothers or carers of children aged 0-2 years. Participants in intervention arms were invited to either face to face or online intervention arms, facilitated by trained multilingual community facilitators (CFs), offering culturally informed discussions on child nutrition and care practices. The qualitative study was embedded within this trial, collecting feedback through interviews and focus groups. Thematic analysis was employed to identify key themes, focusing on intervention fidelity and acceptance.

Results

Of the initial attendees, 42 (from the remaining 153 at the study's conclusion) and 9 CFs offered feedback on the intervention's delivery and suggestions for enhancing community-based interventions' success. Key findings highlighted the need for a more flexible approach to boost participation and the significance of providing accessible, translated documents and resources. Participants expressed a strong preference for a hybrid model of intervention delivery, combining face-to-face and online sessions to accommodate diverse needs.

Conclusion

Parenting interventions, particularly for new mothers, may engage more of the target population by adopting a hybrid design. This would provide attendees with the flexibility to select the delivery method, session timings and the option to participate at any stage of the intervention. The study underscores the importance of cultural tailoring and flexible delivery methods in enhancing participation and engagement in community-based health interventions.

The Challenges for EU User Testing Policies for Patient Information Leaflets

Patient information leaflets (PILs) are essential tools in healthcare, providing crucial information about medication use. In the European Union, the European Medicines Agency (EMA) oversees the regulation and standardisation of PILs to ensure their readability and accessibility. However, challenges persist in ensuring these documents are comprehensible and user-friendly. This study employs a qualitative analytical approach, reviewing existing literature and regulatory documents to identify gaps in the EU user testing policies for PILs. It focuses on the diversity of participant samples, the independence of the testing process, and the robustness of user testing protocols. Findings indicate that current user testing practices often lack diversity and may be biased when pharmaceutical companies conduct their own tests. Additionally, there is a lack of user testing protocols for translated PILs, potentially compromising their accuracy and cultural relevance. To improve the efficacy of PILs, it is essential to include diverse and representative samples in user testing, mandate independent third-party evaluations, implement protocols for user testing on translated PILs, and ensure continuous updates to guidelines based on the latest best practices in health communication. These measures will enhance patient safety and understanding of medication information.

Utilizing nursing standards in electronic health records: A descriptive qualitative study

BACKGROUND

The electronic health record (EHR), including standardized structures and languages, represents an important data source for nurses, to continually update their individual and shared perceptual understanding of clinical situations. Registered nurses' utilization of nursing standards, such as standardized nursing care plans and language in EHRs, has received little attention in the literature. Further research is needed to understand nurses' care planning and documentation practice.

AIMS

This study aimed to describe the experiences and perceptions of nurses' EHR documentation practices utilizing standardized nursing care plans including standardized nursing language, in the daily documentation of nursing care for patients living in special dementia-care units in nursing homes in Norway.

METHODS

A descriptive qualitative study was conducted between April and November 2021 among registered nurses working in special dementia care units in Norwegian nursing homes. In-depth interviews were conducted, and data was analyzed utilizing reflexive thematic analysis with a deductive orientation. Findings Four themes were generated from the analysis. First, the knowledge, skills, and attitude of system users were perceived to influence daily documentation practice. Second, management and organization of documentation work, internally and externally, influenced motivation and engagement in daily documentation processes. Third, usability issues of the EHR were perceived to limit the daily workflow and the nurses' information-needs. Last, nursing standards in the EHR were perceived to contribute to the development of documentation practices, supporting and stimulating ethical awareness, cognitive processes, and knowledge development.

CONCLUSION

Nurses and nursing leaders need to be continuously involved and engaged in EHR documentation to safeguard development and implementation of relevant nursing standards.

Evaluating clinical decision support software (CDSS): challenges for robust evidence generation

OBJECTIVES

Computerized clinical decision support software (CDSS) are digital health technologies that have been traditionally categorized as medical devices. However, the evaluation frameworks for traditional medical devices are not well adapted to assess the value and safety of CDSS. In this study, we identified a range of challenges associated with CDSS evaluation as a medical device and investigated whether and how CDSS are evaluated in Australia.

METHODS

Using a qualitative approach, we interviewed 11 professionals involved in the implementation and evaluation of digital health technologies at national and regional levels. Data were thematically analyzed using both data-driven (inductive) and theory-based (deductive) approaches.

RESULTS

Our results suggest that current CDSS evaluations have an overly narrow perspective on the risks and benefits of CDSS due to an inability to capture the impact of the technology on the sociotechnical environment. By adopting a static view of the CDSS, these evaluation frameworks are unable to discern how rapidly evolving technologies and a dynamic clinical environment can impact CDSS performance. After software upgrades, CDSS can transition from providing information to specifying diagnoses and treatments. Therefore, it is not clear how CDSS can be monitored continuously when changes in the software can directly affect patient safety.

CONCLUSION

Our findings emphasize the importance of taking a living health technology assessment approach to the evaluation of digital health technologies that evolve rapidly. There is a role for observational (real-world) evidence to understand the impact of changes to the technology and the sociotechnical environment on CDSS performance.

A Sustainable Model for Healthcare Systems: The Innovative Approach of ESG and Digital Transformation

In recent years, the globe has faced a series of topics of growing concern, such as the COVID-19 pandemic, the international financial crisis, rising socio-economic inequalities, the negative outcomes of greenhouse gas emissions, which resulted in climate change, and many others. Organizations worldwide have confronted these new challenges of sustainable finance by incorporating environmental, social, and corporate governance (ESG) factors and digital transformation (DT) in their innovation business strategies. The healthcare sector represents a large share of the global economy (about 10% of global economic output), employs a large number of workers, and needs to rely more on an open innovation model where interested parties, especially patients, are going to have a say in their own well-being. Thus, it is imperative that healthcare providers be efficient, effective, resilient, and sustainable in the face of significant challenges and risks. At the same time, they must offer sustainable development goals and digital transformation to healthcare users through limited governmental resources. This study investigates the role, importance, and correlation of ESG factors and digital transformation to the sustainable finance of healthcare systems through an innovative model. The main purpose of the paper is to present the already implemented ESG and DT factors in the healthcare sector and to propose a mutual and combined implementation strategy based on common evaluation tools, methods, and actions. A set of proposed actions and strategies are presented for the sustainability and resilience of the healthcare sector.

Design and Implementation of an Inpatient Fall Risk Management Information System

BACKGROUND

Falls had been identified as one of the nursing-sensitive indicators for nursing care in hospitals. With technological progress, health information systems make it possible for health care professionals to manage patient care better. However, there is a dearth of research on health information systems used to manage inpatient falls.

OBJECTIVE

This study aimed to design and implement a novel hospital-based fall risk management information system (FRMIS) to prevent inpatient falls and improve nursing quality.

METHODS

This implementation was conducted at a large academic medical center in central China. We established a nurse-led multidisciplinary fall prevention team in January 2016. The hospital's fall risk management problems were summarized by interviewing fall-related stakeholders, observing fall prevention workflow and post-fall care process, and investigating patients' satisfaction. The FRMIS was developed using an iterative design process, involving collaboration among health care professionals, software developers, and system architects. We used process indicators and outcome indicators to evaluate the implementation effect.

RESULTS

The FRMIS includes a fall risk assessment platform, a fall risk warning platform, a fall preventive strategies platform, fall incident reporting, and a tracking improvement platform. Since the implementation of the FRMIS, the inpatient fall rate was significantly lower than that before implementation (P<.05). In addition, the percentage of major fall-related injuries was significantly lower than that before implementation. The implementation rate of fall-related process indicators and the reporting rate of high risk of falls were significantly different before and after system implementation (P<.05).

CONCLUSIONS

The FRMIS provides support to nursing staff in preventing falls among hospitalized patients while facilitating process control for nursing managers.

A Data-Driven Paradigm for a Resilient and Sustainable Integrated Health Information Systems for Health Care Applications

Introduction

Many transformations and uncertainties, such as the fourth industrial revolution and pandemics, have propelled healthcare acceptance and deployment of health information systems (HIS). External and internal determinants aligning with the global course influence their deployments. At the epic is digitalization, which generates endless data that has permeated healthcare. The continuous proliferation of complex and dynamic healthcare data is the digitalization frontier in healthcare that necessitates attention.

Objective

This study explores the existing body of information on HIS for healthcare through the data lens to present a data-driven paradigm for healthcare augmentation paramount to attaining a sustainable and resilient HIS.

Method

Preferred Reporting Items for Systematic Reviews and Meta-Analyses: PRISMA-compliant in-depth literature review was conducted systematically to synthesize and analyze the literature content to ascertain the value disposition of HIS data in healthcare delivery.

Results

This study details the aspects of a data-driven paradigm for robust and sustainable HIS for health care applications. Data source, data action and decisions, data sciences techniques, serialization of data sciences techniques in the HIS, and data insight implementation and application are data-driven features expounded. These are essential data-driven paradigm building blocks that need iteration to succeed.

Discussions

Existing literature considers insurgent data in healthcare challenging, disruptive, and potentially revolutionary. This view echoes the current healthcare quandary of good and bad data availability. Thus, data-driven insights are essential for building a resilient and sustainable HIS. People, technology, and tasks dominated prior HIS frameworks, with few data-centric facets. Improving healthcare and the HIS requires identifying and integrating crucial data elements.

Conclusion

The paper presented a data-driven paradigm for a resilient and sustainable HIS. The findings show that data-driven track and components are essential to improve healthcare using data analytics insights. It provides an integrated footing for data analytics to support and effectively assist health care delivery.

Provenance Data Management in Health Information Systems: A Systematic Literature Review

AIMS

This article aims to perform a Systematic Literature Review (SLR) to better understand the structures of different methods, techniques, models, methodologies, and technologies related to provenance data management in health information systems (HISs). The SLR developed here seeks to answer the questions that contribute to describing the results.

METHOD

An SLR was performed on six databases using a search string. The backward and forward snowballing technique was also used. Eligible studies were all articles in English that presented on the use of different methods, techniques, models, methodologies, and technologies related to provenance data management in HISs. The quality of the included articles was assessed to obtain a better connection to the topic studied.

RESULTS

Of the 239 studies retrieved, 14 met the inclusion criteria described in this SLR. In order to complement the retrieved studies, 3 studies were included using the backward and forward snowballing technique, totaling 17 studies dedicated to the construction of this research. Most of the selected studies were published as conference papers, which is common when involving computer science in HISs. There was a more frequent use of data provenance models from the PROV family in different HISs combined with different technologies, among which blockchain and middleware stand out. Despite the advantages found, the lack of technological structure, data interoperability problems, and the technical unpreparedness of working professionals are still challenges encountered in the management of provenance data in HISs.

CONCLUSION

It was possible to conclude the existence of different methods, techniques, models, and combined technologies, which are presented in the proposal of a taxonomy that provides researchers with a new understanding about the management of provenance data in HISs.

Effects of primary healthcare reform on routine health information systems (RHISs): a mixed-methods study in Lesotho

BACKGROUND AND OBJECTIVE

The Ministry of Health of Lesotho and Partners In Health piloted the Lesotho National Primary Health Care Reform (LPHCR) from July 2014 to June 2017 to improve quality and quantity of service delivery and enhance health system management. This initiative included improvement of routine health information systems (RHISs) to map disease burden and reinforce data utilisation for clinical quality improvement.

METHODS AND ANALYSIS

The WHO Data Quality Assurance framework's core indicators were used to compare the completeness of health data before versus after the LPHCR in 60 health centres and 6 hospitals across four districts. To examine change in data completeness, we conducted an interrupted time series analysis using multivariable logistic mixed-effects regression. Additionally, we conducted 25 key informant interviews with healthcare workers (HCWs) at the different levels of Lesotho's health system, following a purposive sampling approach. Interviews were analysed using deductive coding based on the Performance of Routine Information System Management framework, which inspected organisational, technical and behavioural factors influencing RHIS processes and outputs associated with the LPHCR.

RESULTS

In multivariable analyses, trends in monthly data completion rate were higher after versus before the LPHCR for: documenting first antenatal care visit (adjusted OR (AOR): 1.24, 95% CI: 1.14 to 1.36) and institutional delivery (AOR: 1.19, 95% CI: 1.07 to 1.32). When discussing processes, HCWs highlighted the value of establishing clear roles and responsibilities in reporting under a new organisational structure, improved community programmes among district health management teams, and enhanced data sharing and monitoring by districts.

CONCLUSION

The Ministry of Health had a strong data completion rate pre-LPHCR, which was sustained throughout the LPHCR despite increased service utilisation. The data completion rate was optimised through improved behavioural, technical and organisational factors introduced as part of the LPHCR.

Disparities of Health Program Information Systems in Indonesia: A Cross-Sectional Indonesian Health Facility Research 2019

Although a recording and reporting format for health centers already exists for Indonesia's standard information system, numerous health applications still need to meet the needs of each program. Therefore, this study aimed to demonstrate the potential disparities in information systems in the application and data collection of health programs among Indonesian community health centers (CHCs) based on provinces and regions. This cross-sectional research used data from 9831 CHCs from the Health Facilities Research 2019 (RIFASKES). Significance was assessed using a chi-square test and analysis of variance (ANOVA). The number of applications was depicted on a map using the spmap command with STATA version 14. It showed that region 2, which represented Java and Bali, was the best, followed by regions 1, which comprised Sumatra Island and its surroundings, and 3, Nusa Tenggara. The highest mean, equaling that of Java, was discovered in three provinces of region 1, namely, Jambi, Lampung, and Bangka Belitung. Furthermore, Papua and West Papua had less than 60% for all types of data-storage programs. Hence, there is a disparity in the health information system in Indonesia by province and region. The results of this analysis recommend future improvement of the CHCs' information systems.

Evaluation of an Electronic Medical Record Module for Nursing Documentation in Paediatric Palliative Care: Involvement of Nurses with a Think-Aloud Approach

BACKGROUND

Paediatric palliative care (PPC) is a noncurative approach to the care of children and adolescents with life-limiting and life-threatening illnesses. Electronic medical records (EMRs) play an important role in documenting such complex processes. Despite their benefits, they can introduce unintended consequences if future users are not involved in their development.

AIM

The aim of this study was to evaluate the acceptance of a novel module for nursing documentation by nurses working in the context of PPC.

METHODS

An observational study employing concurrent think-aloud and semi-structured qualitative interviews were conducted with 11 nurses working in PPC. Based on the main determinants of the unified theory of acceptance and use of technology (UTAUT), data were analysed using qualitative content analysis.

RESULTS

The main determinants of UTAUT were found to potentially influence acceptance of the novel module. Participants perceived the module to be self-explanatory and intuitive. Some adaptations, such as the reduction of fragmentation in the display, the optimization of confusing mouseover fields, and the use of familiar nursing terminology, are reasonable ways of increasing software adoption.

CONCLUSIONS

After adaptation of the modules based on the results, further evaluation with the participation of future users is required.

Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation

OBJECTIVE

Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation.

METHODS

The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies.

RESULTS

The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment.

DISCUSSION

Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation.

CONCLUSION

As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

Information Technology Ambidexterity, Digital Dynamic Capability, and Knowledge Processes as Enablers of Patient Agility: Empirical Study

BACKGROUND

There is a limited understanding of information technology's (IT) role as an enabler of patient agility and the department's ability to respond to patients' needs and wishes adequately.

OBJECTIVE

This study aims to contribute to the insights of the validity of the hypothesized relationship among IT resources, practices and capabilities, and hospital departments' knowledge processes, and the department's ability to adequately sense and respond to patient needs and wishes (ie, patient agility).

METHODS

This study conveniently sampled data from 107 clinical hospital departments in the Netherlands and used structural equation modeling for model assessment.

RESULTS

IT ambidexterity positively enhanced the development of a digital dynamic capability (β=.69; t4999=13.43; P<.001). Likewise, IT ambidexterity also positively impacted the hospital department's knowledge processes (β=.32; t4999=2.85; P=.005). Both digital dynamic capability (β=.36; t4999=3.95; P<.001) and knowledge processes positively influenced patient agility (β=.33; t4999=3.23; P=.001).

CONCLUSIONS

IT ambidexterity promotes taking advantage of IT resources and experiments to reshape patient services and enhance patient agility.

Barriers and enablers in implementing an electronic incident reporting system in a teaching hospital: A case study from Saudi Arabia

BACKGROUND

Widespread recognition of the impact of healthcare adverse events has triggered incident reporting system implementation to promote patient safety. The aim was to assess the effectiveness, usability, enablers, and barriers of the Electronic Occurrence Variance Reporting System (eOVR) in addition to end user satisfaction.

METHODS

This study comprised a cross-sectional survey two years after implementation of the eOVR. Secondary data analysis evaluated the volume of incident reporting before and after implementing the eOVR.

OUTCOME MEASURES

Primary outcome measures: satisfaction and system usability, system security, workplace safety culture, training, and reporting trends. An overall satisfaction was collected. Secondary outcome: rate of reported OVRs per 1000 admissions. Furthermore, barriers and enablers to the reporting process were explored.

RESULTS

Study findings indicate that the eOVR has been successful in terms of high satisfaction according to respondents. Most of the respondents found the system easy to access, maintained patient confidentiality and reporting anonymity. Around half the respondents indicated having a non-punitive culture of reporting in their hospital. Physicians had significantly lower scores in all primary outcomes Incident reporting increased by 33.6% (p < 0.0001) after implementing the eOVR.

CONCLUSION

Successful incident reporting systems should be easy and simple to use, accessible and include features that guarantee anonymity and confidentiality. End-users should be trained prior to launching such a system. The implementation of such systems needs to be combined with promoting a just culture in the organization, timely feedback, more involvement and focus on physicians and junior staff which will improve user satisfaction and reporting rates.

A multivariate statistical evaluation of actual use of electronic health record systems implementations in Kenya

Background

Health facilities in developing countries are increasingly adopting Electronic Health Records systems (EHRs) to support healthcare processes. However, only limited studies are available that assess the actual use of the EHRs once adopted in these settings. We assessed the state of the 376 KenyaEMR system (national EHRs) implementations in healthcare facilities offering HIV services in Kenya.

Methods

The study focused on seven EHRs use indicators. Six of the seven indicators were programmed and packaged into a query script for execution within each KenyaEMR system (KeEMRs) implementation to collect monthly server-log data for each indicator for the period 2012–2019. The indicators included: Staff system use, observations (clinical data volume), data exchange, standardized terminologies, patient identification, and automatic reports. The seventh indicator (EHR variable Completeness) was derived from routine data quality report within the EHRs. Data were analysed using descriptive statistics, and multiple linear regression analysis was used to examine how individual facility characteristics affected the use of the system.

Results

213 facilities spanning 19 counties participated in the study. The mean number of authorized users who actively used the KeEMRs was 18.1% (SD = 13.1%, p<0.001) across the facilities. On average, the volume of clinical data (observations) captured in the EHRs was 3363 (SD = 4259). Only a few facilities(14.1%) had health data exchange capability. 97.6% of EHRs concept dictionary terms mapped to standardized terminologies such as CIEL. Within the facility EHRs, only 50.5% (SD = 35.4%, p< 0.001) of patients had the nationally-endorsed patient identifier number recorded. Multiple regression analysis indicated the need for improvement on the mode of EHRs use of implementation.

Conclusion

The standard EHRs use indicators can effectively measure EHRs use and consequently determine success of the EHRs implementations. The results suggest that most of the EHRs use areas assessed need improvement, especially in relation to active usage of the system and data exchange readiness.

Evaluation of a guidelines website capitalizing on Finnish content and Belgium interface: A pilot study in French general practice

No reference point-of-care, web-based medical compendium is available in general practice in France. We have then conducted the experimentation of EBMPracticeNet, a Belgian website of guidelines translated and adapted from the Finnish EBM Guidelines. We collected data from three sources: (i) the website logbook; (ii) a search-specific assessment questionnaire; (iii) a global assessment questionnaire. A cumulative number of 262 (62.8%) physicians performed at least one search on the website and clicked on average 5.9 times per month. Physicians globally got an accurate answer (74.2%). They found the information provided by the website reliable (92.2%) and useful for practice (78.6%). They perceived the website ergonomics as good. The main reported barriers were the time and effort required to find an accurate answer and the uneven relevance of the information retrieved. Improvements should focus on guidelines indexing and their adaptation to the French context, and training physicians to search medical databases.

Developing a tool to monitor knowledge translation in the health system: results from an international Delphi study

BACKGROUND

It is generally accepted that evidence-informed decision making contributes to better health system performance and health outcomes, yet we are lacking benchmarks to monitor the impact of national health information systems (HIS) in policy and practice. Hence in this study, we have aimed to identify criteria for monitoring Knowledge Translation (KT) capacity within countries.

METHODS

We conducted a web-based Delphi with over 120 public health professionals from 45 countries to reach agreement on criteria to monitor KT at the level of national HIS. Public health professionals participated in three survey rounds, in which they ranked 85 preselected criteria and could suggest additional criteria.

RESULTS

Experts working in national (public) health agencies and statistical offices, as well as in health policy and care agreed on 29 criteria which constitute the Health Information (HI)-Impact Index. The criteria cover four essential domains of evaluation: the production of high-quality evidence, broad access and dissemination, stakeholder engagement and knowledge integration across sectors and in civil society. The HI-Impact Index was pretested by officials working in ministries of health and public health agencies in eight countries; they found the tool acceptable and user-friendly.

CONCLUSIONS

The HI-Impact Index provides benchmarks to monitor KT so that countries can assess whether high-quality evidence can be easily accessed and used by the relevant stakeholders in health policy and practice, by civil society and across sectors. Next steps include further refining the procedure for conducting the assessment in routine, and sharing experiences from HIS evaluations using the HI-Impact Index.

Routine Health Information Systems in the European Context: A Systematic Review of Systematic Reviews

(1) Background: The aim of this study is to provide a better understanding of the requirements to improve routine health information systems (RHISs) for the management of health systems, including the identification of best practices, opportunities, and challenges in the 53 countries and territories of the WHO European region. (2) Methods: We conducted an overview of systematics reviews and searched the literature in the databases MEDLINE/PubMed, Cochrane, EMBASE, and Web of Science electronic databases. After a meticulous screening, we identified 20 that met the inclusion criteria, and RHIS evaluation results were presented according to the Performance of Routine Information System Management (PRISM) framework. (3) Results: The reviews were published between 2007 and 2020, focusing on the use of different systems or technologies and aimed to analyze interventions on professionals, centers, or patients' outcomes. All reviews examined showed variability in results in accordance with the variability of interventions and target populations. We have found different areas for improvement for RHISs according to the three determinants of the PRISM framework that influence the configuration of RHISs: technical, organizational, or behavioral elements. (4) Conclusions: RHIS interventions in the European region are promising. However, new global and international strategies and the development of tools and mechanisms should be promoted to highly integrate platforms among European countries.

The usability of Jordan stillbirths and neonatal deaths surveillance (JSANDS) system: results of focus group discussions

Background

Jordan Stillbirths and Neonatal Deaths Surveillance system (JSANDS) is a newly developed system and is currently implemented in five large hospitals in Jordan. This study aimed at exploring the healthcare professionals’ perception about the usability of JSANDS.

Methods

A descriptive qualitative approach, using focus group discussions, was adopted. A total of 5 focus groups including 23 focal points were conducted in five participating hospitals in Jordan.

Results

Data analysis identified nine main issues related to the JSANDS system: the system usefulness, the system performance, data quality, the system limitations, human rights, female empowerment, nurses’ competencies strengthened, the sustainability of the JSANDS, and COVID-19 impact on the system. Users reported that JSANDS data were useful, the system was simple and easy to use, and the data were accurate and complete. However, some users reported that some technical issues need to be enhanced.

Conclusions

JSANDS was perceived positively by the current users. According to them, it provides a formative and comprehensive data on stillbirths and neonatal deaths and their causes, and therefore, was recommended to be adopted by its users and scaled up.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13690-021-00551-1.

A Model for Examining Challenges and Opportunities in Use of Cloud Computing for Health Information Systems

Health Information Systems (HIS) are becoming crucial for health providers, not only for keeping Electronic Health Records (EHR) but also because of the features they provide that can be lifesaving, thanks to the advances in Information Technology (IT). These advancements have led to increasing demands for additional features to these systems to improve their intelligence, reliability, and availability. All these features may be provisioned through the use of cloud computing in HIS. This study arrives at three dimensions pertinent to adoption of cloud computing in HIS through extensive interviews with experts, professional expertise and knowledge of one of the authors working in this area, and review of academic and practitioner literature. These dimensions are financial performance and cost; IT operational excellence and DevOps; and security, governance, and compliance. Challenges and drivers in each of these dimensions are detailed and operationalized to arrive at a model for HIS adoption. This proposed model detailed in this study can be employed by executive management of health organizations, especially senior clinical management positions like Chief Technology Officers (CTOs), Chief Information Officers (CIOs), and IT managers to make an informed decision on adoption of cloud computing for HIS. Use of cloud computing to support operational and financial excellence of healthcare organizations has already made some headway in the industry, and its use in HIS would be a natural next step. However, due to the mission′s critical nature and sensitivity of information stored in HIS, the move may need to be evaluated in a holistic fashion that can be aided by the proposed dimensions and the model. The study also identifies some issues and directions for future research for cloud computing adoption in the context of HIS.

Development of standard indicators to assess use of electronic health record systems implemented in low-and medium-income countries

Background

Electronic Health Record Systems (EHRs) are being rolled out nationally in many low- and middle-income countries (LMICs) yet assessing actual system usage remains a challenge. We employed a nominal group technique (NGT) process to systematically develop high-quality indicators for evaluating actual usage of EHRs in LMICs.

Methods

An initial set of 14 candidate indicators were developed by the study team adapting the Human Immunodeficiency Virus (HIV) Monitoring, Evaluation, and Reporting indicators format. A multidisciplinary team of 10 experts was convened in a two-day NGT workshop in Kenya to systematically evaluate, rate (using Specific, Measurable, Achievable, Relevant, and Time-Bound (SMART) criteria), prioritize, refine, and identify new indicators. NGT steps included introduction to candidate indicators, silent indicator ranking, round-robin indicator rating, and silent generation of new indicators. 5-point Likert scale was used in rating the candidate indicators against the SMART components.

Results

Candidate indicators were rated highly on SMART criteria (4.05/5). NGT participants settled on 15 final indicators, categorized as system use (4); data quality (3), system interoperability (3), and reporting (5). Data entry statistics, systems uptime, and EHRs variable concordance indicators were rated highest.

Conclusion

This study describes a systematic approach to develop and validate quality indicators for determining EHRs use and provides LMICs with a multidimensional tool for assessing success of EHRs implementations.

Identifying the Value of an eHealth Intervention Aimed at Cognitive Impairments: Observational Study in Different Contexts and Service Models

BACKGROUND

Value is one of the central concepts in health care, but it is vague within the field of summative eHealth evaluations. Moreover, the role of context in explaining the value is underexplored, and there is no explicit framework guiding the evaluation of the value of eHealth interventions. Hence, different studies conceptualize and operationalize value in different ways, ranging from measuring outcomes such as clinical efficacy or behavior change of patients or professionals to measuring the perceptions of various stakeholders or in economic terms.

OBJECTIVE

The objective of our study is to identify contextual factors that determine similarities and differences in the value of an eHealth intervention between two contexts. We also aim to reflect on and contribute to the discussion about the specification, assessment, and relativity of the "value" concept in the evaluation of eHealth interventions.

METHODS

The study concerned a 6-month eHealth intervention targeted at elderly patients (n=107) diagnosed with cognitive impairment in Italy and Sweden. The intervention introduced a case manager role and an eHealth platform to provide remote monitoring and coaching services to the patients. A model for evaluating the value of eHealth interventions was designed as monetary and nonmonetary benefits and sacrifices, based on the value conceptualizations in eHealth and marketing literature. The data was collected using the Mini-Mental State Examination (MMSE), the clock drawing test, and the 5-level EQ-5D (EQ-5D-5L). Semistructured interviews were conducted with patients and health care professionals. Monetary data was collected from the health care and technology providers.

RESULTS

The value of an eHealth intervention applied to similar types of populations but differed in different contexts. In Sweden, patients improved cognitive performance (MMSE mean 0.85, SD 1.62, P<.001), reduced anxiety (EQ-5D-5L mean 0.16, SD 0.54, P=.046), perceived their health better (EQ-5D-5L VAS scale mean 2.6, SD 9.7, P=.035), and both patients and health care professionals were satisfied with the care. However, the Swedish service model demonstrated an increased cost, higher workload for health care professionals, and the intervention was not cost-efficient. In Italy, the patients were satisfied with the care received, and the health care professionals felt empowered and had an acceptable workload. Moreover, the intervention was cost-effective. However, clinical efficacy and quality of life improvements have not been observed. We identified 6 factors that influence the value of eHealth intervention in a particular context: (1) service delivery design of the intervention (process of delivery), (2) organizational setup of the intervention (ie, organizational structure and professionals involved), (3) cost of different treatments, (4) hourly rates of staff for delivering the intervention, (5) lifestyle habits of the population (eg, how physically active they were in their daily life and if they were living alone or with family), and (6) local preferences on the quality of patient care.

CONCLUSIONS

Value in the assessments of eHealth interventions need to be considered beyond economic terms, perceptions, or behavior changes. To obtain a holistic view of the value created, it needs to be operationalized into monetary and nonmonetary outcomes, categorizing these into benefits and sacrifices.

Exploring the Use of Evidence From the Development and Evaluation of an Electronic Health (eHealth) Trial: Case Study

BACKGROUND

Evidence-based practice refers to building clinical decisions on credible research evidence, professional experience, and patient preferences. However, there is a growing concern that evidence in the context of electronic health (eHealth) is not sufficiently used when forming policies and practice of health care. In this context, using evaluation and research evidence in clinical or policy decisions dominates the discourse. However, the use of additional types of evidence, such as professional experience, is underexplored. Moreover, there might be other ways of using evidence than in clinical or policy decisions.

OBJECTIVE

This study aimed to analyze how different types of evidence (such as evaluation outcomes [including patient preferences], professional experiences, and existing scientific evidence from other research) obtained within the development and evaluation of an eHealth trial are used by diverse stakeholders. An additional aim was to identify barriers to the use of evidence and ways to support its use.

METHODS

This study was built on a case of an eHealth trial funded by the European Union. The project included 4 care centers, 2 research and development companies that provided the web-based physical exercise program and an activity monitoring device, and 2 science institutions. The qualitative data collection included 9 semistructured interviews conducted 8 months after the evaluation was concluded. The data analysis concerned (1) activities and decisions that were made based on evidence after the project ended, (2) evidence used for those activities and decisions, (3) in what way the evidence was used, and (4) barriers to the use of evidence.

RESULTS

Evidence generated from eHealth trials can be used by various stakeholders for decisions regarding clinical integration of eHealth solutions, policy making, scientific publishing, research funding applications, eHealth technology, and teaching. Evaluation evidence has less value than professional experiences to local decision making regarding eHealth integration into clinical practice. Professional experiences constitute the evidence that is valuable to the highest variety of activities and decisions in relation to eHealth trials. When using existing scientific evidence related to eHealth trials, it is important to consider contextual relevance, such as location or disease. To support the use of evidence, it is suggested to create possibilities for health care professionals to gain experience, assess a few rather than a large number of variables, and design for shorter iterative cycles of evaluation.

CONCLUSIONS

Initiatives to support and standardize evidence-based practice in the context of eHealth should consider the complexities in how the evidence is used in order to achieve better uptake of evidence in practice. However, one should be aware that the assumption of fact-based decision making in organizations is misleading. In order to create better chances that the evidence produced would be used, this should be addressed through the design of eHealth trials.

How can we monitor the impact of national health information systems? Results from a scoping review

BACKGROUND

National health information (HI) systems provide data on population health, the determinants of health and health system performance within countries. The evaluation of these systems has traditionally focused on statistical practices and procedures, and not on data use or reuse for policy and practice. This limits the capacity to assess the impact of HI systems on healthcare provision, management and policy-making. On the other hand, the field of Knowledge Translation (KT) has developed frameworks to guide evidence into practice.

METHODS

A scoping review of the KT literature to identify the essential mechanisms and determinants of KT that could help monitor the impact of HI systems.

RESULTS

We examined 79 publications and we identified over 100 different KT frameworks but none of these were focused on HI systems per se. There were specific recommendations on disseminating evidence to stakeholders at the institutional and organizational level, and on sustaining the use of evidence in practice and the broader community setting.

CONCLUSIONS

We developed a new model, the HI-Impact framework, in which four domains are essential for mapping the impact of national HI systems: (i) HI Evidence Quality, (ii) HI System Responsiveness, (iii) Stakeholder Engagement and (iv) Knowledge Integration. A comprehensive impact assessment of HI systems requires addressing the use of HI in public health decision-making, health service delivery and in other sectors which might have not been considered previously. Monitoring Stakeholder Engagement and Knowledge Integration certifies that the use of HI in all policies is an explicit point of assessment.

Doing More Than Asking for Opinions

Health information systems (HIS) are used to manage information related to population health. The goal of this research was to conduct an evaluation of a HIS used at a hospital in south-western Uganda using participatory approaches. The evaluation structure was based on guidelines generated by the Center for Disease Control and Prevention and a series of theoretical and methodological concepts regarding participatory engagement that encouraged stakeholder participation throughout the evaluation. The primary objectives were to describe the areas of strength and limitations of the HIS, and develop potential system enhancements. Ultimately, engagement of local staff members throughout each stage of the evaluation resulted in the development of a series of recommendations considered relevant and feasible by local stakeholders. We build on these results by highlighting the value of stakeholder engagement and opportunities to apply participatory and community-based research methods and an Ecohealth framework to an HIS evaluation.

Developing and Applying a Formative Evaluation Framework for Health Information Technology Implementations: Qualitative Investigation

BACKGROUND

There is currently a lack of comprehensive, intuitive, and usable formative evaluation frameworks for health information technology (HIT) implementations. We therefore sought to develop and apply such a framework. This study describes the Technology, People, Organizations, and Macroenvironmental factors (TPOM) framework we developed.

OBJECTIVE

The aim was to develop and apply a formative evaluation framework for HIT implementations, highlighting interrelationships between identified dimensions and offering guidance for implementers.

METHODS

We drew on an initial prototype framework developed as part of a literature review exploring factors for the effective implementation of HIT. In addition, we used qualitative data from three national formative evaluations of different HIT interventions (electronic health record, electronic prescribing, and clinical decision support functionality). The combined data set comprised 19 case studies of primarily hospital settings, and included 703 semistructured interviews, 663 hours of observations, and 864 documents gathered from a range of care settings across National Health Service (NHS) England and NHS Scotland. Data analysis took place over a period of 10 years and was guided by a framework informed by the existing evidence base.

RESULTS

TPOM dimensions are intimately related and each include a number of subthemes that evaluators need to consider. Although technological functionalities are crucial in getting an initiative off the ground, system design needs to be cognizant of the accompanying social and organizational transformations required to ensure that technologies deliver the desired value for a variety of stakeholders. Wider structural changes, characterized by shifting policy landscapes and markets, influence technologies and the ways they are used by organizations and staff.

CONCLUSIONS

The TPOM framework supports formative evaluations of HIT implementation and digitally enabled transformation efforts. There is now a need for prospective application of the TPOM framework to determine its value.

[Opinion of doctors and users on e-health advances in primary care]

OBJECTIVE

To find out the opinions of users and primary care doctors on the progress of e-health in primary care.

DESIGN

Focus groups set up between 1st May and 25th September 2018.

LOCATION

Primary care level in regional online health services.

PARTICIPANTS

A total of 29 users and 33 primary care doctors linked to primary care that were contacted, through user associations and regional public health services, respectively.

METHOD

Inductive analysis of transcripts from focus groups about the following issues: accessibility and security; efficiency; information and technical problems; potentialities.

RESULTS

Online health services are positively valued in terms of cost savings both by users and doctors of primary care. Nevertheless, face-to-face interaction is still considered a key factor in the quality of service. When it comes to the access and provision of e-health services, there is shared concern for the security of the data and for the training deficiencies of users and doctors. An adequate design of the applications becomes a condition for the acceptance and diffusion of technologies that support online health.

CONCLUSIONS

Health authorities should intensify the use of new technologies oriented towards easing the work of doctors, simplifying bureaucratic tasks, providing diagnostic and prescription security, and protecting data. As long as systems generate failures and mistrust, new technologies will remain to be perceived as a complement, but not as an alternative to the traditional care model.

Human-Centered Design Strategies for Device Selection in mHealth Programs: Development of a Novel Framework and Case Study

BACKGROUND

Despite the increasing use of remote measurement technologies (RMT) such as wearables or biosensors in health care programs, challenges associated with selecting and implementing these technologies persist. Many health care programs that use RMT rely on commercially available, "off-the-shelf" devices to collect patient data. However, validation of these devices is sparse, the technology landscape is constantly changing, relative benefits between device options are often unclear, and research on patient and health care provider preferences is often lacking.

OBJECTIVE

To address these common challenges, we propose a novel device selection framework extrapolated from human-centered design principles, which are commonly used in de novo digital health product design. We then present a case study in which we used the framework to identify, test, select, and implement off-the-shelf devices for the Remote Assessment of Disease and Relapse-Central Nervous System (RADAR-CNS) consortium, a research program using RMT to study central nervous system disease progression.

METHODS

The RADAR-CNS device selection framework describes a human-centered approach to device selection for mobile health programs. The framework guides study designers through stakeholder engagement, technology landscaping, rapid proof of concept testing, and creative problem solving to develop device selection criteria and a robust implementation strategy. It also describes a method for considering compromises when tensions between stakeholder needs occur.

RESULTS

The framework successfully guided device selection for the RADAR-CNS study on relapse in multiple sclerosis. In the initial stage, we engaged a multidisciplinary team of patients, health care professionals, researchers, and technologists to identify our primary device-related goals. We desired regular home-based measurements of gait, balance, fatigue, heart rate, and sleep over the course of the study. However, devices and measurement methods had to be user friendly, secure, and able to produce high quality data. In the second stage, we iteratively refined our strategy and selected devices based on technological and regulatory constraints, user feedback, and research goals. At several points, we used this method to devise compromises that addressed conflicting stakeholder needs. We then implemented a feedback mechanism into the study to gather lessons about devices to improve future versions of the RADAR-CNS program.

CONCLUSIONS

The RADAR device selection framework provides a structured yet flexible approach to device selection for health care programs and can be used to systematically approach complex decisions that require teams to consider patient experiences alongside scientific priorities and logistical, technical, or regulatory constraints.

What quality factors matter in enhancing the perceived benefits of online health information sites? Application of the updated DeLone and McLean Information Systems Success Model

BACKGROUND

Despite a growing need for designing and monitoring health information sites through comprehensive examination of the various elements of website quality, there is little research that systematically models and presents such examinations.

OBJECTIVES

Applying the updated DeLone and McLean Model of Information Systems Success, this research aimed to examine how health information sites' information quality, system quality, and service quality lead to user satisfaction and perceived benefits.

METHODS

This research was conducted in a specific context of the National Health Information Portal (NHIP), a governmental health information site in South Korea. We conducted online survey in 2017, with 506 adults from the NHIP consumer panel. Data were analyzed using a confirmatory factor analysis, hierarchical ordinary least squares regression, and bootstrapping approach for a mediation test.

RESULTS

Of the three quality factors, information quality had significant associations with all outcome variables: user satisfaction, intention to reuse the site, and perceived benefits of site use in health settings. There were also indirect paths from information quality to perceived benefits, one mediated through intention and the other mediated through satisfaction and then intention. Service quality had a significant association with user satisfaction, and its impact on perceived benefits occurred indirectly through user satisfaction and intention in serial. By contrast, the role of system quality received no empirical support.

IMPLICATIONS

The results offer theoretical and practical implications for how to enhance the effectiveness of online health information sites.

A Model for Evaluating Big Data Analytics Tools for Organisation Purposes

Big data analytics tools have many functions that are common or similar to one another. This is a problem for many organisations that are either interested or have deployed some of the tools. The problem arises because there is no mechanism on how to determine appropriateness of the tools within context in an environment. Some of the implications of the problem are that it is difficult to assess appropriateness of the tools in an environment, which sometimes result to duplication; and the value of the tools. The aim of this project was to propose a solution through a model that can be used to evaluate big data analytics tools for organisations’ benefits. The qualitative method, case study approach and semi-structured interview technique were applied in the study. From the analysis, the role of criteria, business and IT alignment, governance, and skill-sets were revealed as critical factors. Based on the factors, a model was developed, which can be used as a building block through which evaluation of big data analytics tools in an organisation is carried out.

Evaluation of the Integrated Health Information System (IHIS) in Public Hospitals in Cyprus Utilizing the DIPSA Framework

INTRODUCTION

The Ministry of Health of the Republic of Cyprus has implemented an Integrated Health Information System (IHIS) in two hospitals. However, no evaluation of IHIS has been conducted to assess its safety, efficiency and effectiveness. The proper utilization of IHIS is essential for the provision of quality healthcare services.

AIM

The purpose of this study was to evaluate the current IHIS in public hospitals in Cyprus utilizing the DIPSA evaluation framework.

METHODS

A total of 309 subjects, including doctors, nurses and other healthcare professionals, participated in the study. The DIPSA evaluation framework assessed the users' perception in five categories namely, satisfaction, collaboration, system quality, safety and procedures, using Likert scale and 3 open questions. Correlation between the categories was assessed using the Pearson correlation coefficient, and multiple regression analysis was used to examine the relationship between the demographic characteristics and categories. Data analysis was done using SPSS v24.

RESULTS

All five categories were rated moderately, between 2.5 and 3, by the participants. All categories were correlated (P < 0.01). Multiple regression analysis indicated the need for improvement between the professionals (mainly doctors and nurses) and the categories. The open questions pointed out the need for improvement in all 3 factors examined (Technology, Human Factor, Organization).

DISCUSSION

The moderately rated categories, in the Cyprus IHIS, suggest that there is a lot of room for improvement. Some interventions are suggested that could positively and simultaneously affect one or more categories.

Staff expectations for the implementation of an electronic health record system: a qualitative study using normalisation process theory

BACKGROUND

Global evidence suggests a range of benefits for introducing electronic health record (EHR) systems to improve patient care. However, implementing EHR within healthcare organisations is complex and, in the United Kingdom (UK), uptake has been slow. More research is needed to explore factors influencing successful implementation. This study explored staff expectations for change and outcome following procurement of a commercial EHR system by a large academic acute NHS hospital in the UK.

METHODS

Qualitative interviews were conducted with 14 members of hospital staff who represented a variety of user groups across different specialities within the hospital. The four components of Normalisation Process Theory (Coherence, Cognitive participation, Collective action and Reflexive monitoring) provided a theoretical framework to interpret and report study findings.

RESULTS

Health professionals had a common understanding for the rationale for EHR implementation (Coherence). There was variation in willingness to engage with and invest time into EHR (Cognitive participation) at an individual, professional and organisational level. Collective action (whether staff feel able to use the EHR) was influenced by context and perceived user-involvement in EHR design and planning of the implementation strategy. When appraising EHR (Reflexive monitoring), staff anticipated short and long-term benefits. Staff perceived that quality and safety of patient care would be improved with EHR implementation, but that these benefits may not be immediate. Some staff perceived that use of the system may negatively impact patient care. The findings indicate that preparedness for EHR use could mitigate perceived threats to the quality and safety of care.

CONCLUSIONS

Health professionals looked forward to reaping the benefits from EHR use. Variations in level of engagement suggest early components of the implementation strategy were effective, and that more work was needed to involve users in preparing them for use. A clearer understanding as to how staff groups and services differentially interact with the EHR as they go about their daily work was required. The findings may inform other hospitals and healthcare systems on actions that can be taken prior to EHR implementation to reduce concerns for quality and safety of patient care and improve the chance of successful implementation.

External validation of the unified model of information systems continuance (UMISC): An international comparison

CONTEXT

The Unified Model of Information Systems Continuance (UMISC) is a metamodel for the evaluation of clinical information systems (CISs) that integrates constructs from five models that have previously been published in the literature. UMISC was developed at the Georges Pompidou University Hospital (HEGP) in Paris and was partially validated at the Saint Joseph Hospital Group (HPSJ), another acute care institution using the same CIS as HEGP.

OBJECTIVE

The aim of this replication study was twofold: (1) to perform an external validation of UMISC in two different hospitals and country contexts: the Italian Hospital of Buenos Aires (HIBA) in Argentina and the Hospital Sirio Libanes in Sao Paulo, Brazil (HSL); (2) to compare, using the same evaluation model, the determinants of satisfaction, use, and continuance intention observed at HIBA and HSL with those previously observed at HEGP and HPSJ.

METHODS

The UMISC evaluation questionnaires were translated from their original languages (English and French) to Brazilian Portuguese and Spanish following the translation/back-translation method. These questionnaires were then applied at each target site. The 21 UMISC-associated hypotheses were tested using structural equation modeling (SEM).

RESULTS

A total of 3020 users, 1079 at HIBA and 1941 at the HSL, were included in the analysis. The respondents included 1406 medical staff and 1001 nursing staff. The average profession-adjusted use, overall satisfaction and continuance intention were significantly higher at HIBA than at HSL in the medical and nursing groups. In SEM analysis, UMISC explained 23% and 11% of the CIS use dimension, 72% and 85% of health professionals' satisfaction, and 41% and 60% of continuance intention at HIBA and HSL, respectively. Twenty of the 21 UMISC-related hypotheses were validated in at least one of the four evaluation sites, and 16 were validated in two or more sites.

CONCLUSION

The UMISC evaluation metamodel appears to be a robust comparison and explanatory model of satisfaction, use and continuance intention for CISs in late post adoption situations.

Experience of implementing new mental health indicators within information systems in six low- and middle-income countries

BACKGROUND

Successful scale-up of integrated primary mental healthcare requires routine monitoring of key programme performance indicators. A consensus set of mental health indicators has been proposed but evidence on their use in routine settings is lacking.

AIMS

To assess the acceptability, feasibility, perceived costs and sustainability of implementing indicators relating to integrated mental health service coverage in six South Asian (India, Nepal) and sub-Saharan African countries (Ethiopia, Nigeria, South Africa, Uganda).

METHOD

A qualitative study using semi-structured key informant interviews (n = 128) was conducted. The 'Performance of Routine Information Systems' framework served as the basis for a coding framework covering three main categories related to the performance of new tools introduced to collect data on mental health indicators: (1) technical; (2) organisation; and (3) behavioural determinants.

RESULTS

Most mental health indicators were deemed relevant and potentially useful for improving care, and therefore acceptable to end users. Exceptions were indicators on functionality, cost and severity. The simplicity of the data-capturing formats contributed to the feasibility of using forms to generate data on mental health indicators. Health workers reported increasing confidence in their capacity to record the mental health data and minimal additional cost to initiate mental health reporting. However, overstretched primary care staff and the time-consuming reporting process affected perceived sustainability.

CONCLUSIONS

Use of the newly developed, contextually appropriate mental health indicators in health facilities providing primary care services was seen largely to be feasible in the six Emerald countries, mainly because of the simplicity of the forms and continued support in the design and implementation stage. However, approaches to implementation of new forms generating data on mental health indicators need to be customised to the specific health system context of different countries. Further work is needed to identify ways to utilise mental health data to monitor and improve the quality of mental health services.

DECLARATION OF INTEREST

None.

An effective density-based clustering and dynamic maintenance framework for evolving medical data streams

BACKGROUND

Medical data stream clustering has become an integral part of medical decision systems since it extracts highly-sensitive information from a tremendous flow of medical data. However, clustering and maintaining of medical data streams is still a challenging task. That is because the evolving of medical data streams imposes various challenges for clustering such as the ability to discover the arbitrary shape of a cluster, the ability to group data streams without a predefined number of clusters, and the ability to maintain the data clusters dynamically.

OBJECTIVE

To support the online medical decisions, there is a need to address the clustering challenges. Therefore, in this paper, we propose an effective density-based clustering and dynamic maintenance framework for grouping the patients with similar symptoms into meaningful clusters and monitoring the patients' status frequently.

METHODS

For clustering, we generate a set of initial medical data clusters based on the combination of Piece-wise Aggregate Approximation and the density-based spatial clustering of applications with noise called (PAA+DBSCAN) algorithm. For maintenance, when new medical data streams arrive, we maintain the initially generated medical data clusters dynamically. Since the incremental cluster maintenance is time-consuming, we further propose an Advanced Cluster Maintenance (ACM) approach to improve the performance of the dynamic cluster maintenance.

RESULTS

The experimental results on real-world medical datasets demonstrate the effectiveness and efficiency of our proposed approaches. The PAA+DBSCAN algorithm is more efficient and effective than the exact DBSCAN algorithm. Moreover, the ACM approach requires less running time in comparison with the Baseline Cluster Maintenance (BCM) approach using different tuning parameter values in all datasets. That is because the BCM approach tracks all the data points in the cluster.

CONCLUSION

The proposed framework is capable of clustering and maintaining the medical data streams effectively by means of grouping the patients who share similar symptoms and tracking the patients status that naturally tends to be changing over time.

The Iterative Convergent Design for Mobile Health Usability Testing: Mixed Methods Approach

Although patients express an interest in using mobile health (mHealth) interventions to manage their health and chronic conditions, many current mHealth interventions are difficult to use. Usability testing is critical for the success of novel mHealth interventions. Researchers recognize the utility of using qualitative and quantitative approaches for usability testing, but many mHealth researchers lack the awareness of integration approaches from advances in mixed methods research that can add value to mHealth technology. As efficient usability testing proceeds iteratively, we introduce a novel mixed methods design developed specifically for mHealth researchers. The iterative convergent mixed methods design involves simultaneous qualitative and quantitative data collection and analysis that continues cyclically through multiple rounds of mixed methods data collection and analysis until the mHealth technology under evaluation is found to work to the satisfaction of the researcher. In cyclical iterations, early development is more qualitatively driven but progressively becomes more quantitatively driven. Using this design, mHealth researchers can leverage mixed methods integration procedures in the research question, data collection, data analysis, interpretation, and dissemination dimensions. This study demonstrates how the iterative convergent mixed methods design provides a novel framework for generating unique insights into multifaceted phenomena impacting mHealth usability. Understanding these practices can help developers and researchers leverage the strengths of an integrated mixed methods design.

Optimizing medication use through a synergistic technology testing process integrating implementation science to drive effectiveness and facilitate scale

OBJECTIVES

1) To describe a synergistic technology testing process (STTP) that integrates traditional technology assessment with implementation science principles to drive uptake, enhance outcomes, and facilitate scaling of medication optimization health information technology solutions; and 2) to illustrate the application of the STTP using an example that involves designing and testing a medication therapy problem (MTP) platform for use by pharmacists in primary care.

SUMMARY

Optimizing medication services requires supportive technologies that have been fully tested before release. Current testing approaches are not sufficient to produce the information needed to accelerate uptake and drive impact. Implementation science principles can supplement the traditional testing process by broadening its focus to include designing a truly usable technology, attending to contextual influences, studying the implementation process, and assessing the technology for its scalability. The STTP is an early attempt at outlining the integration of traditional technology testing with implementation science for pharmacy practice.

CONCLUSION

The potential impact of technology-supported medication optimization solutions to improve patient outcomes, enhance quality of care, and reduce costs could be substantial. Accelerating uptake, driving impact, and facilitating scaling will require innovative testing paradigms that result in evidence-based technologies that can feasibly be implemented in real-world settings.

Management information systems for community based interventions to improve health: qualitative study of stakeholder perspectives

BACKGROUND

Community based providers are well place to deliver behavioural interventions to improve health. Good project management and reliable outcome data are needed to efficiently deliver and evaluate such interventions, and Management information systems (MIS) can facilitate these processes. We explored stakeholders perspectives on the use of MIS in community based behavioural interventions.

METHODS

Stakeholders, purposively selected to provide a range of MIS experience in the delivery of community based behavioural interventions to improve health (public health commissioners, intervention service managers, project officers, health researchers and MIS designers), were invited to participate in individual semi-structured interviews. We used a topic guide and encouraged stakeholders to reflect on their experiences.: Interviews were recorded, transcribed and analysed using five steps of Framework analysis. We applied an agreed coding framework and completed the interviews when no new themes emerged.

RESULTS

We interviewed 15 stakeholders. Key themes identified were: (i) MIS access; (ii) data and its function; (iii) MIS development and updating. Within these themes the different experiences, needs, use, training and expertise of stakeholders and the variation and potential of MIS were evidenced. Interviews advised the need to involve stakeholders in MIS design and development, build-in flexibility to accommodate MIS refinement and build on effective MIS.

CONCLUSIONS

Findings advised involving stakeholders, early in the design process. Designs should build on existing MIS of proven utility and ensure flexibility in the design, to incorporate adaptations and ongoing system development in response to early MIS use and evolving stakeholder needs.