Evaluation of medium-term consequences of implementing commercial computerized physician order entry and clinical decision support prescribing systems in two 'early adopter' hospitals

Essential strategic principles for planning and developing digitally enabled interventions in health and care settings

Health and care information technologies can improve service delivery, but unfortunately many fail to realise their potential. This is because tools fail to effectively integrate with user and organisational practices. We here draw on two decades and an extensive body of practical and theoretical experience of conducting formative evaluations in health service settings to extract seven essential principles for planning and developing digital tools. Firstly, it is important to determine if digitalisation is the right approach for the desired change and to view these tools as complex interventions needing careful planning. The development process should embrace varying levels of co-creation and adapt to different needs and contexts. Ensuring that tools integrate well with existing information infrastructures and striking a balance between being widely applicable and locally relevant is essential. Furthermore, planning for how these interventions will fit into broader health and care pathways is vital. By considering these factors, decision-makers can significantly improve the chances of successful implementation, adoption and scaling of health and care technologies.

Applying the Non-Adoption, Abandonment, Scale-up, Spread, and Sustainability Framework Across Implementation Stages to Identify Key Strategies to Facilitate Clinical Decision Support System Integration Within a Large Metropolitan Health Service: Interview and Focus Group Study

BACKGROUND

Computerized clinical decision support systems (CDSSs) enhance patient care through real-time, evidence-based guidance for health care professionals. Despite this, the effective implementation of these systems for health services presents multifaceted challenges, leading to inappropriate use and abandonment over the course of time. Using the Non-Adoption, Abandonment, Scale-Up, Spread, and Sustainability (NASSS) framework, this qualitative study examined CDSS adoption in a metropolitan health service, identifying determinants across implementation stages to optimize CDSS integration into health care practice.

OBJECTIVE

This study aims to identify the theory-informed (NASSS) determinants, which included multiple CDSS interventions across a 2-year period, both at the health-service level and at the individual hospital setting, that either facilitate or hinder the application of CDSSs within a metropolitan health service. In addition, this study aimed to map these determinants onto specific stages of the implementation process, thereby developing a system-level understanding of CDSS application across implementation stages.

METHODS

Participants involved in various stages of the implementation process were recruited (N=30). Participants took part in interviews and focus groups. We used a hybrid inductive-deductive qualitative content analysis and a framework mapping approach to categorize findings into barriers, enablers, or neutral determinants aligned to NASSS framework domains. These determinants were also mapped to implementation stages using the Active Implementation Framework stages approach.

RESULTS

Participants comprised clinical adopters (14/30, 47%), organizational champions (5/30, 16%), and those with roles in organizational clinical informatics (5/30, 16%). Most determinants were mapped to the organization level, technology, and adopter subdomains. However, the study findings also demonstrated a relative lack of long-term implementation planning. Consequently, determinants were not uniformly distributed across the stages of implementation, with 61.1% (77/126) identified in the exploration stage, 30.9% (39/126) in the full implementation stage, and 4.7% (6/126) in the installation stages. Stakeholders engaged in more preimplementation and full-scale implementation activities, with fewer cycles of monitoring and iteration activities identified.

CONCLUSIONS

These findings addressed a substantial knowledge gap in the literature using systems thinking principles to identify the interdependent dynamics of CDSS implementation. A lack of sustained implementation strategies (ie, training and longer-term, adopter-level championing) weakened the sociotechnical network between developers and adopters, leading to communication barriers. More rigorous implementation planning, encompassing all 4 implementation stages, may, in a way, help in addressing the barriers identified and enhancing enablers.

Using Theories, Models, and Frameworks to Inform Implementation Cycles of Computerized Clinical Decision Support Systems in Tertiary Health Care Settings: Scoping Review

BACKGROUND

Computerized clinical decision support systems (CDSSs) are essential components of modern health system service delivery, particularly within acute care settings such as hospitals. Theories, models, and frameworks may assist in facilitating the implementation processes associated with CDSS innovation and its use within these care settings. These processes include context assessments to identify key determinants, implementation plans for adoption, promoting ongoing uptake, adherence, and long-term evaluation. However, there has been no prior review synthesizing the literature regarding the theories, models, and frameworks that have informed the implementation and adoption of CDSSs within hospitals.

OBJECTIVE

This scoping review aims to identify the theory, model, and framework approaches that have been used to facilitate the implementation and adoption of CDSSs in tertiary health care settings, including hospitals. The rationales reported for selecting these approaches, including the limitations and strengths, are described.

METHODS

A total of 5 electronic databases were searched (CINAHL via EBSCOhost, PubMed, Scopus, PsycINFO, and Embase) to identify studies that implemented or adopted a CDSS in a tertiary health care setting using an implementation theory, model, or framework. No date or language limits were applied. A narrative synthesis was conducted using full-text publications and abstracts. Implementation phases were classified according to the "Active Implementation Framework stages": exploration (feasibility and organizational readiness), installation (organizational preparation), initial implementation (initiating implementation, ie, training), full implementation (sustainment), and nontranslational effectiveness studies.

RESULTS

A total of 81 records (42 full text and 39 abstracts) were included. Full-text studies and abstracts are reported separately. For full-text studies, models (18/42, 43%), followed by determinants frameworks (14/42,33%), were most frequently used to guide adoption and evaluation strategies. Most studies (36/42, 86%) did not list the limitations associated with applying a specific theory, model, or framework.

CONCLUSIONS

Models and related quality improvement methods were most frequently used to inform CDSS adoption. Models were not typically combined with each other or with theory to inform full-cycle implementation strategies. The findings highlight a gap in the application of implementation methods including theories, models, and frameworks to facilitate full-cycle implementation strategies for hospital CDSSs.

Optimising ePrescribing in hospitals through the interoperability of systems and processes: a qualitative study in the UK, US, Norway and the Netherlands

BACKGROUND

Investment in the implementation of hospital ePrescribing systems has been a priority in many economically-developed countries in order to modernise the delivery of healthcare. However, maximum gains in the safety, quality and efficiency of care are unlikely to be fully realised unless ePrescribing systems are further optimised in a local context. Typical barriers to optimal use are often encountered in relation to a lack of systemic capacity and preparedness to meet various levels of interoperability requirements, including at the data, systems and services levels. This lack of systemic interoperability may in turn limit the opportunities and benefits potentially arising from implementing novel digital heath systems.

METHODS

We undertook n = 54 qualitative interviews with key stakeholders at nine digitally advanced hospital sites across the UK, US, Norway and the Netherlands. We included hospitals featuring 'standalone, best of breed' systems, which were interfaced locally, and multi-component and integrated electronic health record enterprise systems. We analysed the data inductively, looking at strategies and constraints for ePrescribing interoperability within and beyond hospital systems.

RESULTS

Our thematic analysis identified 4 main drivers for increasing ePrescribing systems interoperability: (1) improving patient safety (2) improving integration & continuity of care (3) optimising care pathways and providing tailored decision support to meet local and contextualised care priorities and (4) to enable full patient care services interoperability in a variety of settings and contexts. These 4 interoperability dimensions were not always pursued equally at each implementation site, and these were often dependent on the specific national, policy, organisational or technical contexts of the ePrescribing implementations. Safety and efficiency objectives drove optimisation targeted at infrastructure and governance at all levels. Constraints to interoperability came from factors such as legacy systems, but barriers to interoperability of processes came from system capability, hospital policy and staff culture.

CONCLUSIONS

Achieving interoperability is key in making ePrescribing systems both safe and useable. Data resources exist at macro, meso and micro levels, as do the governance interventions necessary to achieve system interoperability. Strategic objectives, most notably improved safety, often motivated hospitals to push for evolution across the entire data architecture of which they formed a part. However, hospitals negotiated this terrain with varying degrees of centralised coordination. Hospitals were heavily reliant on staff buy-in to ensure that systems interoperability was built upon to achieve effective data sharing and use. Positive outcomes were founded on a culture of agreement about the usefulness of access by stakeholders, including prescribers, policymakers, vendors and lab technicians, which was reflected in an alignment of governance goals with system design.

Impact of computerised provider order entry on the quality and quantity of clinical information included with investigation requests: an interrupted time series analysis

INTRODUCTION

Relevant clinical information is vital to inform the analytical and interpretative phases of most investigations. The aim of this study is to evaluate the impact of implementation of computerised provider order entry (CPOE), featuring order-specific electronic order entry forms (eOEFs), on the quality and quantity of clinical information included with investigation requests.

METHODS

The CPOE module of a commercially available electronic health record (Cerner Millennium) was implemented at a large, tertiary care centre. The laboratory information management system was interrogated to collect data on specimens sent for microbiological culture 1 year before implementation of CPOE (2018), immediately post implementation (2019) and 6 months post implementation (2020). An interrupted time series analysis was performed, using text mining, to evaluate the quality and quantity of free-text clinical information.

RESULTS

In total, 39 919 specimens were collected from 16 458 patients. eOEFs were used to place 10 071 out of 13 735 orders in 2019 (73.3%), and 9155 out of 12 229 orders in 2020 (74.9%). No clinical details were included with 653 out of 39 919 specimens (1.6%), of which 22 (3.4%) were ordered using eOEFs. The median character count increased from 14 in 2018, to 41 in 2019, and 38 in 2020. An anti-infective agent was specified in 581 out of 13 955 requests (4.2%) in 2018; 5545 out of 13 735 requests (40.4%) in 2019; and 5215 out of 12 229 requests (42.6%) in 2020. Ciprofloxacin or piperacillin-tazobactam (Tazocin) were mentioned in the clinical details included with 421 out of 15 335 urine culture requests (2.7%), of which 406 (96.3%) were ordered using eOEFs. Subsequent detection of in vitro non-susceptibility led to a change in anti-infective therapy for five patients.

CONCLUSIONS

Implementation of CPOE, featuring order-specific eOEFs, significantly and sustainably improves the quality and quantity of clinical information included with investigation requests, resulting in changes to patient management that would not otherwise have occurred.

Teaching Digital Medicine in a Virtual Classroom: Impacts on Student Mindset and Competencies

Digital competencies, as well as knowledge about digital medicine, are becoming increasingly relevant but are rarely reflected in teaching concepts at universities. One reason for this is probably the fact that they are not yet part of the curricula in many areas or countries (such as medicine in Germany). Therefore, courses that address digital competencies and intend to impart knowledge about digital medicine are not subject to any curricular specifications and have a correspondingly broad range of possible designs. This article reports findings from an investigation on an interdisciplinary and cross-faculty course on digital medicine. An online questionnaire was used to assess student attitudes toward digital medicine topics and conduct self-assessments of their digital competencies before and after the course. The aim of this study was to test whether such a course could influence students' attitudes and competencies. Group comparisons revealed statistically significant changes. They proved that the described course and its content regarding digital transformation in healthcare and digital medicine had an impact on digital competencies and participant opinions on digital topics. In order to teach students important competencies for the 21st century, universities should offer more courses that address digital transformation and support students in improving their competencies.

The Understanding of Medical Informatics among Dental and Medical Students

Background

Current medical care deeply relies on informatics during all stages of patient care, which is significantly enhanced due to its use. The healthcare professional's formation in medical informatics results crucial for their everyday practice. However, healthcare study programs not always provide education about the use of this wide variety of systems, and young professionals find that they need to learn about it over the experience. The aim of this study was to assess the understanding of medical and dental students regarding medical informatics and ICTs.

Materials and Methods

A questionnaire was produced with 3 sections and a total of 24 questions. Students replied to the survey before and after taking the medical informatics course.

Results

A total of 719 students from second year of medical and dental school were recruited for the study between the period of September of 2017-May 2018, September 2018-May 2019, September 2019-May 2020, and September 2020-May 2021. Medical and dental students showed a good level of understanding regarding medical informatics, as well as a good perception of the relevance of ICT learning for the professional practice. Course attendance increased the percentage of students that felt confident of their knowledge about medical informatics. However, most students felt that little or no medical informatics education was lectured at their schools and that the University should adapt the academic program to include it. After taking the course, the student's perception on this matter was improved.

Conclusion

Medical and dental students find medical informatics learning useful for their future professional practice and feel inclined to use it. However, they feel that Universities need to adapt their programs in order to include medical education courses and trainings; partly because they are not completely aware of the use of ICTs that already are established in their courses.

Electronic Medication Management Systems: Analysis of Enhancements to Reduce Errors and Improve Workflow

BACKGROUND

Electronic medication management (eMM) has been shown to reduce medication errors; however, new safety risks have also been introduced that are associated with system use. No research has specifically examined the changes made to eMM systems to mitigate these risks.

OBJECTIVES

To (1) identify system-related medication errors or workflow blocks that were the target of eMM system updates, including the types of medications involved, and (2) describe and classify the system enhancements made to target these risks.

METHODS

In this retrospective qualitative study, documents detailing updates made from November 2014 to December 2019 to an eMM system were reviewed. Medication-related updates were classified according to "rationale for changes" and "changes made to the system."

RESULTS

One hundred and seventeen updates, totaling 147 individual changes, were made to the eMM system over the 4-year period. The most frequent reasons for changes being made to the eMM were to prevent medication errors (24% of reasons), optimize workflow (22%), and support "work as done" on paper (16%). The most frequent changes made to the eMM were options added to lists (14% of all changes), extra information made available on the screen (8%), and the wording or phrasing of text modified (8%). Approximately a third of the updates (37%) related to high-risk medications. The reasons for system changes appeared to vary over time, as eMM functionality and use expanded.

CONCLUSION

To our knowledge, this is the first study to systematically review and categorize system updates made to overcome new safety risks associated with eMM use. Optimization of eMM is an ongoing process, which changes over time as users become more familiar with the system and use is expanded to more sites. Continuous monitoring of the system is necessary to detect areas for improvement and capitalize on the benefits an electronic system can provide.

Guidance for publishing qualitative research in informatics

Qualitative research, the analysis of nonquantitative and nonquantifiable data through methods such as interviews and observation, is integral to the field of biomedical and health informatics. To demonstrate the integrity and quality of their qualitative research, authors should report important elements of their work. This perspective article offers guidance about reporting components of the research, including theory, the research question, sampling, data collection methods, data analysis, results, and discussion. Addressing these points in the paper assists peer reviewers and readers in assessing the rigor of the work and its contribution to the literature. Clearer and more detailed reporting will ensure that qualitative research will continue to be published in informatics, helping researchers disseminate their understanding of people, organizations, context, and sociotechnical relationships as they relate to biomedical and health data.

Physicians' Use of the Computerized Physician Order Entry System for Medication Prescribing: Systematic Review

BACKGROUND

Computerized physician order entry (CPOE) systems in health care settings have many benefits for prescribing medication, such as improved quality of patient care and patient safety. However, to achieve their full potential, the factors influencing the usage of CPOE systems by physicians must be identified and understood.

OBJECTIVE

The aim of this study is to identify the factors influencing the usage of CPOE systems by physicians for medication prescribing in their clinical practice.

METHODS

We conducted a systematic search of the literature on this topic using four databases: PubMed, CINAHL, Ovid MEDLINE, and Embase. Searches were performed from September 2019 to December 2019. The retrieved papers were screened by examining the titles and abstracts of relevant studies; two reviewers screened the full text of potentially relevant papers for inclusion in the review. Qualitative, quantitative, and mixed methods studies with the aim of conducting assessments or investigations of factors influencing the use of CPOE for medication prescribing among physicians were included. The identified factors were grouped based on constructs from two models: the unified theory of acceptance and use of technology model and the Delone and McLean Information System Success Model. We used the Mixed Method Appraisal Tool to assess the quality of the included studies and narrative synthesis to report the results.

RESULTS

A total of 11 articles were included in the review, and 37 factors related to the usage of CPOE systems were identified as the factors influencing how physicians used CPOE for medication prescribing. These factors represented three main themes: individual, technological, and organizational.

CONCLUSIONS

This study identified the common factors that influenced the usage of CPOE systems by physicians for medication prescribing regardless of the type of setting or the duration of the use of a system by participants. Our findings can be used to inform implementation and support the usage of the CPOE system by physicians.

Usability evaluation of an integrated electronic medication management system implemented in an oncology setting using the unified theory of the acceptance and use of technology

BACKGROUND

Medication management processes in an Oncology setting are complex and difficult to examine in isolation from interrelated processes and contextual factors. This qualitative study aims to evaluate the usability of an Electronic Medication Management System (EMMS) implemented in a specialised oncology unit using the Unified Theory of Acceptance and Use of Technology (UTAUT) framework.

METHODS

The study was conducted in a 12-bed outpatient Oncology unit of a major teaching hospital 6 months following implementation of a commercial EMMS. In-depth semi-structured interviews were conducted with doctors, nurses and pharmacists using the system to assess usability. The UTAUT framework was used to analyse the results, which facilitated evaluation of interrelated aspects and provided a structured summary of user experience and usability factors.

RESULTS

Direct cross-comparison between user groups illustrated that doctors and pharmacists were generally satisfied with the facilitating conditions (hardware and training), but had divergent perceptions of performance (automation, standardised protocols and communication and documented) and effort (mental and temporal demand) expectancy. In counterpoint, nurses were generally satisfied across all constructs. Prior experience using an alternative EMMS influenced performance and effort expectancy and was related to early dissatisfaction with the EMMS. Furthermore, whilst not originally designed for the healthcare setting, the flexibility of the UTAUT allowed for translation to the hospital environment.

CONCLUSION

Nurses demonstrated overall satisfaction with the EMMS, whilst doctors and pharmacists perceived usability problems, particularly related to restricted automaticity and system complexity, which hindered perceived EMMS success. The study demonstrates the feasibility and utility of the UTAUT framework to evaluate usability of an EMMS for multiple user groups in the Oncology setting.

Overview of digital health teaching courses in medical education in Germany in 2020

Objective: The digitalization of the healthcare system poses new challenges for physicians. Thus, the relevance of learning digital competencies (DiCo), such as dealing with data sets, apply telemedicine or using apps, is already growing in medical education. DiCo should be clearly separated from digitized teaching formats, which have been increasingly used since the COVID 19 pandemic. This article outlines the faculties in Germany where DiCo are already integrated into medical education. Methods: Courses with DiCo as teaching content were collected by a literature research on Pubmed and Google as well as by contacting all dean's offices and other persons responsible for teaching at German medical faculties. The courses were summarized in a table. Results: In a first survey, 16 universities were identified that offer courses on DiCo. In the elective area at the universities, 17 courses and in the compulsory area eight courses could be identified. The scope and content of the courses diverged between compulsory curricula, integrated courses of different lengths, and elective courses that are one-time or longitudinally integrated. The topics taught are heterogeneous and include fundamentals of medical informatics such as data management on the one hand and a collection of e.g. ethics, law, apps, artificial intelligence, telemedicine and robotics on the other hand. Conclusion: Currently, only some German medical faculties offer courses on DiCo. These courses vary in scope and design. They are frequently part of the elective curriculum and only reach some of the students. The possibility of embedding DiCo in the already existing cross-sectional area appears limited. In view of the ongoing digitalization of healthcare, it is necessary to make future courses on DiCo accessible to all medical students. In order to drive this expansion forward, the implementation of the new learning objectives catalogue, in which DiCo are integrated, a network formation, a teaching qualification as well as the involvement of students is recommended.

Medication errors related to computerized provider order entry systems in hospitals and how they change over time: A narrative review

BACKGROUND

Evaluations of computerized provider order entry (CPOE) systems have revealed that reductions in certain types of medication errors occur simultaneously with the emergence of system-related errors - errors that are unlikely or not possible to occur with the use of paper-based medication charts. System-related errors appear to persist many years post-implementation of CPOE, although little is known about whether the types and rates of system-related errors that occur immediately following CPOE implementation are similar to those that endure or emerge after years of system use.

OBJECTIVE

To analyze and synthesize the literature on system-related errors, specifically in relation to the length of time that CPOE systems have been in use, to determine what is currently known about how system-related errors change over time.

METHODS

A literature search was undertaken using the PubMed database to identify English language articles published between January 2005 and March 2020 that provided original data on system-related errors resulting from CPOE system use. Studies were included if they provided results on system-related errors and information relating to the length of time that CPOE had been in use.

RESULTS

Thirty-one studies met the inclusion criteria for this narrative review. System-related errors were identified and described during short, medium and long-term use of CPOE systems, but no single study examined how errors changed over time. In comparing findings across studies, results suggest that system-related errors persist with long-term use of CPOE systems, although likely to occur at a reduced rate.

CONCLUSIONS

This review has highlighted a significant gap in knowledge on how system-related errors change over time. Determining what and when system-related errors occur and the system factors that contribute to their occurrence at different time points after CPOE implementation is necessary for the future prevention and mitigation of these errors.

Clinical Decision Support May Link Multiple Domains to Improve Patient Care: Viewpoint

Integrating clinical decision support (CDS) across the continuum of population-, encounter-, and precision-level care domains may improve hospital and clinic workflow efficiency. Due to the diversity and volume of electronic health record data, complexity of medical and operational knowledge, and specifics of target user workflows, the development and implementation of comprehensive CDS is challenging. Additionally, many providers have an incomplete understanding of the full capabilities of current CDS to potentially improve the quality and efficiency of care delivery. These varied requirements necessitate a multidisciplinary team approach to CDS development for successful integration. Here, we present a practical overview of current and evolving applications of CDS approaches in a large academic setting and discuss the successes and challenges. We demonstrate that implementing CDS tools in the context of linked population-, encounter-, and precision-level care provides an opportunity to integrate complex algorithms at each level into a unified mechanism to improve patient management.

Formative independent evaluation of a digital change programme in the English National Health Service: study protocol for a longitudinal qualitative study

INTRODUCTION

Many countries are launching large-scale, digitally enabled change programmes as part of efforts to improve the quality, safety and efficiency of care. We have been commissioned to conduct an independent evaluation of a major national change programme, the Global Digital Exemplar (GDE) Programme, which aims to develop exemplary digital health solutions and encourage their wider adoption by creating a learning ecosystem across English National Health Service (NHS) provider organisations.

METHODS AND ANALYSIS

This theoretically informed, qualitative, longitudinal formative evaluation comprises five inter-related work packages. We will conduct a combination of 12 in-depth and 24 broader qualitative case studies in GDE sites exploring digital transformation, local learning and mechanisms of spread of knowledge within the Programme and across the wider NHS. Data will be collected through a combination of semistructured interviews with managers, implementation staff (clinical and non-clinical), vendors and policymakers, plus non-participant observations of meetings, site visits, workshops and documentary analysis of strategic local and national plans. Data will be analysed through inductive and deductive methods, beginning with in-depth case study sites and testing the findings against data from the wider sample and national stakeholders.

ETHICS AND DISSEMINATION

This work is commissioned as part of a national change programme and is therefore a service evaluation. We have ethical approval from the University of Edinburgh. Results will be disseminated at six monthly intervals to national policymakers, and made available via our publicly accessible website. We will also identify lessons for the management and evaluation of large-scale evolving digital health change programmes that are of international relevance.

Digital Health in Physicians' and Pharmacists' Office: A Comparative Study of e-Prescription Systems' Architecture and Digital Security in Eight Countries

e-Prescription systems are key components and drivers of digital health. They can enhance the safety of the patients, and are gaining popularity in health care systems around the world. Yet, there is little knowledge on comparative international analysis of e-Prescription systems' architecture and digital security. We report, in this study, original findings from a comparative analysis of the e-Prescription systems in eight different countries, namely, Canada, United States, United Kingdom, Australia, Spain, Japan, Sweden, and Denmark. We surveyed the databases related to pharmacies, eHealth, e-Prescriptions, and related digital health websites for each country, and their system architectures. We also compared the digital security and privacy protocols in place within and across these digital systems. We evaluated the systems' authentication protocols used by pharmacies to verify patients' identities during the medication dispensing process. Furthermore, we examined the supporting systems/services used to manage patients' medication histories and enhance patients' medication safety. Taken together, we report, in this study, original comparative findings on the limitations and challenges of the surveyed systems as well as in adopting e-Prescription systems. While the present study was conducted before the onset of COVID-19, e-Prescription systems have become highly relevant during the current pandemic and hence, a deeper understanding of the country systems' architecture and digital security that can help design effective strategies against the pandemic. e-Prescription systems can help reduce physical contact and the risk of exposure to the virus, as well as the wait times in pharmacies, thus enhancing patient safety and improving planetary health.

Developing and Applying a Formative Evaluation Framework for Health Information Technology Implementations: Qualitative Investigation

BACKGROUND

There is currently a lack of comprehensive, intuitive, and usable formative evaluation frameworks for health information technology (HIT) implementations. We therefore sought to develop and apply such a framework. This study describes the Technology, People, Organizations, and Macroenvironmental factors (TPOM) framework we developed.

OBJECTIVE

The aim was to develop and apply a formative evaluation framework for HIT implementations, highlighting interrelationships between identified dimensions and offering guidance for implementers.

METHODS

We drew on an initial prototype framework developed as part of a literature review exploring factors for the effective implementation of HIT. In addition, we used qualitative data from three national formative evaluations of different HIT interventions (electronic health record, electronic prescribing, and clinical decision support functionality). The combined data set comprised 19 case studies of primarily hospital settings, and included 703 semistructured interviews, 663 hours of observations, and 864 documents gathered from a range of care settings across National Health Service (NHS) England and NHS Scotland. Data analysis took place over a period of 10 years and was guided by a framework informed by the existing evidence base.

RESULTS

TPOM dimensions are intimately related and each include a number of subthemes that evaluators need to consider. Although technological functionalities are crucial in getting an initiative off the ground, system design needs to be cognizant of the accompanying social and organizational transformations required to ensure that technologies deliver the desired value for a variety of stakeholders. Wider structural changes, characterized by shifting policy landscapes and markets, influence technologies and the ways they are used by organizations and staff.

CONCLUSIONS

The TPOM framework supports formative evaluations of HIT implementation and digitally enabled transformation efforts. There is now a need for prospective application of the TPOM framework to determine its value.

Theoretical and methodological considerations in evaluating large-scale health information technology change programmes

BACKGROUND

Attempts to achieve digital transformation across the health service have stimulated increasingly large-scale and more complex change programmes. These encompass a growing range of functions in multiple locations across the system and may take place over extended timeframes. This calls for new approaches to evaluate these programmes.

MAIN BODY

Drawing on over a decade of conducting formative and summative evaluations of health information technologies, we here build on previous work detailing evaluation challenges and ways to tackle these. Important considerations include changing organisational, economic, political, vendor and markets necessitating tracing of evolving networks, relationships, and processes; exploring mechanisms of spread; and studying selected settings in depth to understand local tensions and priorities.

CONCLUSIONS

Decision-makers need to recognise that formative evaluations, if built on solid theoretical and methodological foundations, can help to mitigate risks and help to ensure that programmes have maximum chances of success.

Use of Normalization Process Theory to explore key stakeholders' perceptions of the facilitators and barriers to implementing electronic systems for medicines management in hospital settings

BACKGROUND

Limited data exist on the facilitators and barriers to implementing electronic systems for medicines management in hospitals. Whilst numerous studies advocate system use in improved patient safety and efficiency within the health service, their rate of adoption in practice has been slow.

OBJECTIVE

To explore the perceptions of key stakeholders towards the facilitators and barriers to implementing electronic prescribing systems, robotic pharmacy systems, and automated medication storage and retrieval systems in public hospital settings using Normalization Process Theory as a theoretical framework.

METHODS

Individual face-to-face semi-structured interviews were conducted in three public hospitals in Ireland with 23 consenting participants: nine nurses; four pharmacists; two pharmacy technicians; six doctors; and two Information Technology managers.

RESULTS

Enhanced patient safety and efficiency in healthcare delivery emerged as key facilitators to system implementation, as well as the need to have clinical champions and a multi-disciplinary implementation team to promote engagement and cognitive participation. Key barriers included inadequate training and organisational support, and the need for ease and confidence in system use to achieve collective action.

CONCLUSIONS

Many themes that are potentially transferable to other national settings have been identified and extend the evidence base. This will assist organisations around the world to better plan for implementation of medication-related eHealth systems.

Electronic health records in ambulances: the ERA multiple-methods study

Background

Ambulance services have a vital role in the shift towards the delivery of health care outside hospitals, when this is better for patients, by offering alternatives to transfer to the emergency department. The introduction of information technology in ambulance services to electronically capture, interpret, store and transfer patient data can support out-of-hospital care.

Objective

We aimed to understand how electronic health records can be most effectively implemented in a pre-hospital context in order to support a safe and effective shift from acute to community-based care, and how their potential benefits can be maximised.

Design and setting

We carried out a study using multiple methods and with four work packages: (1) a rapid literature review; (2) a telephone survey of all 13 freestanding UK ambulance services; (3) detailed case studies examining electronic health record use through qualitative methods and analysis of routine data in four selected sites consisting of UK ambulance services and their associated health economies; and (4) a knowledge-sharing workshop.

Results

We found limited literature on electronic health records. Only half of the UK ambulance services had electronic health records in use at the time of data collection, with considerable variation in hardware and software and some reversion to use of paper records as services transitioned between systems. The case studies found that the ambulance services’ electronic health records were in a state of change. Not all patient contacts resulted in the generation of electronic health records. Ambulance clinicians were dealing with partial or unclear information, which may not fit comfortably with the electronic health records. Ambulance clinicians continued to use indirect data input approaches (such as first writing on a glove) even when using electronic health records. The primary function of electronic health records in all services seemed to be as a store for patient data. There was, as yet, limited evidence of electronic health records’ full potential being realised to transfer information, support decision-making or change patient care.

Limitations

Limitations included the difficulty of obtaining sets of matching routine data for analysis, difficulties of attributing any change in practice to electronic health records within a complex system and the rapidly changing environment, which means that some of our observations may no longer reflect reality.

Conclusions

Realising all the benefits of electronic health records requires engagement with other parts of the local health economy and dealing with variations between providers and the challenges of interoperability. Clinicians and data managers, and those working in different parts of the health economy, are likely to want very different things from a data set and need to be presented with only the information that they need.

Future work

There is scope for future work analysing ambulance service routine data sets, qualitative work to examine transfer of information at the emergency department and patients’ perspectives on record-keeping, and to develop and evaluate feedback to clinicians based on patient records.

Study registration

This study is registered as Health and Care Research Wales Clinical Research Portfolio 34166.

Funding

This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 8, No. 10. See the NIHR Journals Library website for further project information.

University of California Technology Wellness Index: A Physician-Centered Framework to Assess Technologies' Impact on Physician Well-Being

Technology is increasingly being incorporated into the everyday workflows of physicians. There are concerns that electronic medical records and other digital technologies will contribute to the growing epidemic of physician burnout. However, some technologies, such as telemedicine, have demonstrated positive effects on physician health by saving time, enhancing work-life balance, improving quality, and restoring more control and flexibility to their practices. Organizations often lack data to evaluate the impact of technologies on physician health. The University of California Technology Wellness Index is a framework that provides a fast, systematic, physician-centered method to assess the impact of technology on physician well-being.

How do stakeholders experience the adoption of electronic prescribing systems in hospitals? A systematic review and thematic synthesis of qualitative studies

Background

Electronic prescribing (ePrescribing) or computerised provider/physician order entry (CPOE) systems can improve the quality and safety of health services, but the translation of this into reduced harm for patients remains unclear. This review aimed to synthesise primary qualitative research relating to how stakeholders experience the adoption of ePrescribing/CPOE systems in hospitals, to help better understand why and how healthcare organisations have not yet realised the full potential of such systems and to inform future implementations and research.

Methods

We systematically searched 10 bibliographic databases and additional sources for citation searching and grey literature, with no restriction on date or publication language. Qualitative studies exploring the perspectives/experiences of stakeholders with the implementation, management, use and/or optimisation of ePrescribing/CPOE systems in hospitals were included. Quality assessment combined criteria from the Critical Appraisal Skills Programme Qualitative Checklist and the Standards for Reporting Qualitative Research guidelines. Data were synthesised thematically.

Results

79 articles were included. Stakeholders’ perspectives reflected a mixed set of positive and negative implications of engaging in ePrescribing/CPOE as part of their work. These were underpinned by further-reaching change processes. Impacts reported were largely practice related rather than at the organisational level. Factors affecting the implementation process and actions undertaken prior to implementation were perceived as important in understanding ePrescribing/CPOE adoption and impact.

Conclusions

Implementing organisations and teams should consider the breadth and depth of changes that ePrescribing/CPOE adoption can trigger rather than focus on discrete benefits/problems and favour implementation strategies that: consider the preimplementation context, are responsive to (and transparent about) organisational and stakeholder needs and agendas and which can be sustained effectively over time as implementations develop and gradually transition to routine use and system optimisation.

The usability aspects of medication-related decision support systems in the inpatient setting: A systematic review

Effort has been made to study the effect of medication-related clinical decision support systems in the inpatient setting; however, there is not much known about the usability of these systems. The goal of this study is to systematically review studies that focused on the usability aspects such as effectiveness, efficiency, and satisfaction of these systems. We systematically searched relevant articles in Scopus, Embase, and PubMed from 1 January 2000 to 1 January 2016, and found 22 articles. Based on Van Welie’s usability model, we categorized usability aspects in terms of usage indicators and means. Our results showed that evidence was mainly found for effectiveness and efficiency. They showed positive results in the usage indicators errors and safety and performance speed. The means warnings and adaptability also had mostly positive results. To date, the effects satisfaction of clinical decision support system remains understudied. Aspects such as memorability, learnability, and consistency require more attention.

Evaluating a handheld decision support device in pediatric intensive care settings

OBJECTIVE

To evaluate end-user acceptance and the effect of a commercial handheld decision support device in pediatric intensive care settings. The technology, pac2, was designed to assist nurses in calculating medication dose volumes and infusion rates at the bedside.

MATERIALS AND METHODS

The devices, manufactured by InformMed Inc., were deployed in the pediatric and neonatal intensive care units in 2 health systems. This mixed methods study assessed end-user acceptance, as well as pac2's effect on the cognitive load associated with bedside dose calculations and the rate of administration errors. Towards this end, data were collected in both pre- and postimplementation phases, including through ethnographic observations, semistructured interviews, and surveys.

RESULTS

Although participants desired a handheld decision support tool such as pac2, their use of pac2 was limited. The nature of the critical care environment, nurses' risk perceptions, and the usability of the technology emerged as major barriers to use. Data did not reveal significant differences in cognitive load or administration errors after pac2 was deployed.

DISCUSSION AND CONCLUSION

Despite its potential for reducing adverse medication events, the commercial standalone device evaluated in the study was not used by the nursing participants and thus had very limited effect. Our results have implications for the development and deployment of similar mobile decision support technologies. For example, they suggest that integrating the technology into hospitals' existing IT infrastructure and employing targeted implementation strategies may facilitate nurse acceptance. Ultimately, the usability of the design will be essential to reaping any potential benefits.

Using electronic patient records: defining learning outcomes for undergraduate education

BACKGROUND

Healthcare professionals are required to access, interpret and generate patient data in the digital environment, and use this information to deliver and optimise patient care. Healthcare students are rarely exposed to the technology, or given the opportunity to use this during their training, which can impact on the digital competence of the graduating workforce. In this study we set out to develop and define domains of competence and associated learning outcomes needed by healthcare graduates to commence working in a digital healthcare environment.

METHOD

A National Working Group was established in the UK to integrate Electronic Patient Records (EPRs) into undergraduate education for healthcare students studying medicine, pharmacy, nursing and midwifery. The working group, comprising 12 academic institutions and representatives from NHS England, NHS Digital and EPR system providers, met to discuss and document key learning outcomes required for using EPRs in the healthcare environment. Outcomes were grouped into six key domains and refined by the group prior to external review by experts working in medical education or with EPRs.

RESULTS

Six key domains of competence and associated learning outcomes were identified and defined. External expert review provided iterative refinement and amendment. The agreed domains were: 1) Digital Health: work as a practitioner in the digital healthcare environment; 2) Accessing Data: access and interpret patient data to inform clinical decision-making; 3) Communication: communicate effectively with healthcare professionals and patients in the digital environment; 4) Generating data: generate data for and about patients within the EPR; 5) Multidisciplinary working: work with healthcare professionals with and alongside EPRs; and 6) Monitoring and audit: monitor and improve the quality and safety of healthcare.

CONCLUSION

The six domains of competence and associated learning outcomes can be used by academics to guide the integration of EPRs into undergraduate healthcare programmes. This is key to ensuring that the future healthcare workforce can work with and alongside EPRs.

Improving medication-related clinical decision support

PURPOSE

Current uses of medication-related clinical decision support (CDS) and recommendations for improving these systems are reviewed.

SUMMARY

Using a systematic approach, articles published from 2007 through 2014 were identified in MEDLINE and EMBASE using MeSH terms and keywords relating to the 5 basic medication-related CDS functionalities. A total of 156 full-text articles and 28 conference abstracts were reviewed across each of the 5 areas: drug-drug interaction (DDI) checks (n = 78), drug allergy checks (n = 20), drug dose support (n = 55), drug duplication checks (n = 11), and drug formulary support (n = 20). The success of medication-related CDS depends on users finding the alerts valuable and acting on the information received. Improving alert specificity and sensitivity is important for all domains. Tiering is important for improving the acceptance of DDI alerts. The ability to perform appropriate cross-sensitivity checks is key to producing appropriate drug allergy checks. Drug dosage alerts should be individualized and deliver practical recommendations. How the system is configured to identify certain drug duplications is important to prevent possible patient toxicity. Accurate knowledge databases are needed to produce relevant drug formulary alerts and encourage formulary adherence. Medication-related CDS is still relatively immature in some organizations and has substantial room for improvement. For example, decision support should consider more patient-specific factors, human factors principles should always be considered, and alert specificity must be improved in order to reduce alert fatigue.

CONCLUSION

Standardization, integration of patient-specific parameters, and consideration of human factors design principles are central to realizing the potential benefits of medication-related CDS.

Evaluation of computerized provider order entry systems: assessing the usability of systems for electronic prescription

BACKGROUND

The field of medicine has been influenced by the growth and development of information systems such as the Computerized Provider Order Entry (CPOE) System.

OBJECTIVE

This study aimed to evaluate the usability of CPOE systems for electronic prescription in Tehran, Iran.

METHODS

This was an evaluation study conducted in 2017. The research population consisted of the CPOE systems used in hospitals of Tehran (Iran) and nurses who had access to, and used, the CPOE systems. Five hospitals with CPOE systems were included in the research sample. The data were collected using a questionnaire, and included a total of 50 questions. The questionnaires were distributed among 254 nurses who were the users of the systems. Data analysis was performed by IBM-SPSS version 21, using independent-samples t-test. A p-value of ≤0.05 was considered statistically significant.

RESULTS

Among the four aspects assessed, the "user-friendliness" (3.87±0.59) had the highest mean score. The lowest mean score (2.01±0.58) was related to the "decision support" feature of the systems. The highest and lowest mean scores for "prescription support" criterion belonged to system E (3.26±0.23) and system C (1.90±0.16), respectively. There was a statistically significant difference between the usability of the systems used in the private and the public hospitals (p<0.001). It was found that the CPOE systems in private hospitals had a higher level of usability (3.42+0.10) compared to those in public hospitals (2.91+0.25).

CONCLUSION

Two main functions of the studied CPOE systems i.e., decision support and prescription support should be developed to make electronic prescription safer and more intuitive. Addressing usability aspects of CPOE systems in practice could improve the usability of these systems for prescription.

Mixed methods study of medication-related decision support alerts experienced during electronic prescribing for inpatients at an English hospital

Objectives

Electronic prescribing and medication administration systems are being introduced in many hospitals worldwide, with varying degrees of clinical decision support including pop-up alerts. Previous research suggests that prescribers override a high proportion of alerts, but little research has been carried out in the UK. Our objective was to explore rates of alert overriding in different prescribing situations and prescribers’ perceptions around the use of decision support alerts in a UK hospital.

Methods

We conducted a mixed methods study on three cardiology wards, directly observing medical and non-medical prescribers’ alert override rates during both ward round and non-ward round prescribing; observations were followed by semi-structured interviews with prescribers, which were then transcribed and analysed thematically.

Results

Overall, 69% of 199 observed alerts were overridden. Alerts experienced during ward rounds were significantly more likely to be overridden than those outside of ward rounds (80% of 56 vs 51% of 63; p=0.001, Χ² test). While respondents acknowledged that alerts could be useful, several also described negative unintended consequences. Many were of the view that usefulness of alerts was limited if the alert was reminding them to do something they would do anyway, or suggesting something they did not feel was relevant. Findings suggest that targeting, timing and additional features of alerts are critical factors in determining whether they are acted on or overridden.

Conclusion

The majority of alerts were overridden. Alerts may be less likely to be overridden if they are built into the prescribing workflow.

Learning health systems need to bridge the ‘two cultures’ of clinical informatics and data science

Background

UK health research policy and plans for population health management are predicated upon transformative knowledge discovery from operational ‘Big Data’. Learning health systems require not only data, but feedback loops of knowledge into changed practice. This depends on knowledge management and application, which in turn depends upon effective system design and implementation. Biomedical informatics is the interdisciplinary field at the intersection of health science, social science and information science and technology that spans this entire scope.

Issues

In the UK, the separate worlds of health data science (bioinformatics, ‘Big Data’) and effective healthcare system design and implementation (clinical informatics, ‘Digital Health’) have operated as ‘two cultures’. Much National Health Service and social care data is of very poor quality. Substantial research funding is wasted on ‘data cleansing’ or by producing very weak evidence. There is not yet a sufficiently powerful professional community or evidence base of best practice to influence the practitioner community or the digital health industry.

Recommendation

The UK needs increased clinical informatics research and education capacity and capability at much greater scale and ambition to be able to meet policy expectations, address the fundamental gaps in the discipline’s evidence base and mitigate the absence of regulation. Independent evaluation of digital health interventions should be the norm, not the exception.

Conclusions

Policy makers and research funders need to acknowledge the existing gap between the ‘two cultures’ and recognise that the full social and economic benefits of digital health and data science can only be realised by accepting the interdisciplinary nature of biomedical informatics and supporting a significant expansion of clinical informatics capacity and capability.

Impact of a commercial order entry system on prescribing errors amenable to computerised decision support in the hospital setting: a prospective pre-post study

Background

In this UK study, we investigated the impact of computerised physician order entry (CPOE) and clinical decision support (CDS) implementation on the rate of 78 high-risk prescribing errors amenable to CDS.

Methods

We conducted a preintervention/postintervention study in three acute hospitals in England. A predefined list of prescribing errors was incorporated into an audit tool. At each site, approximately 4000 prescriptions were reviewed both pre-CPOE and 6 months post-CPOE implementation. The number of opportunities for error and the number of errors that occurred were collated. Error rates were then calculated and compared between periods, as well as by the level of CDS.

Results

The prescriptions of 1244 patients were audited pre-CPOE and 1178 post-CPOE implementation. A total of 28 526 prescriptions were reviewed, with 21 138 opportunities for error identified based on 78 defined errors. Across the three sites, for those prescriptions where opportunities for error were identified, the error rate was found to reduce significantly post-CPOE implementation, from 5.0% to 4.0% (P<0.001). CDS implementation by error type was found to differ significantly between sites, ranging from 0% to 88% across clinical contraindication, dose/frequency, drug interactions and other error types (P<0.001). Overall, 43/78 (55%) of the errors had some degree of CDS implemented in at least one of the hospitals.

Conclusions

Implementation of CPOE with CDS was associated with clinically important reductions in the rate of high-risk prescribing errors. Given the pre-post design, these findings however need to be interpreted with caution. The occurrence of errors was found to be highly dependent on the level of restriction of CDS presented to the prescriber, with the effect that different configurations of the same CPOE system can produce very different results.

Evidence-based Health Informatics: How Do We Know What We Know?

Background: Health IT is expected to have a positive impact on the quality and efficiency of health care. But reports on negative impact and patient harm continue to emerge. The obligation of health informatics is to make sure that health IT solutions provide as much benefit with as few negative side effects as possible. To achieve this, health informatics as a discipline must be able to learn, both from its successes as well as from its failures.

Objectives: To present motivation, vision, and history of evidence-based health informatics, and to discuss achievements, challenges, and needs for action.

Methods: Reflections on scientific literature and on own experiences.

Results: Eight challenges on the way towards evidence-based health informatics are identified and discussed: quality of studies; publication bias; reporting quality; availability of publications; systematic reviews and meta-analysis; training of health IT evaluation experts; translation of evidence into health practice; and post-market surveil-lance. Identified needs for action comprise: establish health IT study registers; increase the quality of publications; develop a taxonomy for health IT systems; improve indexing of published health IT evaluation papers; move from meta-analysis to meta-summaries; include health IT evaluation competencies in curricula; develop evidence-based implementation frameworks; and establish post-marketing surveillance for health IT.

Conclusions: There has been some progress, but evidence-based health informatics is still in its infancy. Building evidence in health informatics is our obligation if we consider medical informatics a scientific discipline.

A systematic review of the types and causes of prescribing errors generated from using computerized provider order entry systems in primary and secondary care

Objective

To understand the different types and causes of prescribing errors associated with computerized provider order entry (CPOE) systems, and recommend improvements in these systems.

Materials and Methods

We conducted a systematic review of the literature published between January 2004 and June 2015 using three large databases: the Cumulative Index to Nursing and Allied Health Literature, Embase, and Medline. Studies that reported qualitative data about the types and causes of these errors were included. A narrative synthesis of all eligible studies was undertaken.

Results

A total of 1185 publications were identified, of which 34 were included in the review. We identified 8 key themes associated with CPOE-related prescribing errors: computer screen display, drop-down menus and auto-population, wording, default settings, nonintuitive or inflexible ordering, repeat prescriptions and automated processes, users' work processes, and clinical decision support systems. Displaying an incomplete list of a patient's medications on the computer screen often contributed to prescribing errors. Lack of system flexibility resulted in users employing error-prone workarounds, such as the addition of contradictory free-text comments. Users' misinterpretations of how text was presented in CPOE systems were also linked with the occurrence of prescribing errors.

Discussion and Conclusions

Human factors design is important to reduce error rates. Drop-down menus should be designed with safeguards to decrease the likelihood of selection errors. Development of more sophisticated clinical decision support, which can perform checks on free-text, may also prevent errors. Further research is needed to ensure that systems minimize error likelihood and meet users' workflow expectations.

Linking individual-level data on diagnoses and dispensing for research on antibiotic use: Evaluation of a novel data source from English secondary care

PURPOSE

There has been a focus on stewardship programmes to curb inappropriate antibiotic prescribing and reduce antimicrobial resistance. In-hospital, patient-level prescribing linked to indication is needed to support surveillance, evaluation of stewardship initiatives, as well as other antibiotic research. We evaluated whether a novel dataset linking hospital pharmacy records to Hospital Episode Statistics data can be used for antibiotic research.

METHODS

Using the Hospital Treatment Insights (HTI) database, which links Hospital Episode Statistics to pharmacy records from 43 out of 153 hospital trusts in England, we estimated the proportion of missed linkage and identified characteristics associated with missing data.

RESULTS

Linkage of antibiotics to patients was inconsistent and dependent on drug type and clinical setting, so that linkage for some specific antibiotics was high (80-100%), but overall, only 27.6% (CI: 27.4%-27.8%) for all antibiotics dispensed. Linkage was best for quinolones (62.6%; CI: 61.8%-63.8%), but only 21.1% (CI: 21.1%-21.2%) for penicillins. Linkage was lower for common antibiotics and in emergency departments; however, 80% linkage was achieved for individual drugs like clindamycin, especially on wards with reduced ward stock use.

CONCLUSIONS

For those antibiotics with high linkage, HTI might be used to study associations between indication, dispensing, and outcomes. However, the majority of common antibiotics had insufficient linkage, likely due to extensive use of ward stocks. Therefore, HTI in its current form is not suitable for general antibiotic surveillance or evaluation of stewardship initiatives. For drugs in HTI other than antibiotics, linkage should be similarly evaluated before a study is conducted.

Longitudinal study of user experiences of a CPOE system in a pediatric hospital

OBJECTIVE

To explore the views of nurses and doctors during the early stages of implementation of a computerized provider order entry (CPOE) system in a pediatric hospital and to examine changes in perceptions and reported behaviors as use of the CPOE system became routine.

METHODS

Semi-structured interviews were undertaken at four time points following CPOE implementation: during week one, week three, week six and then six months following implementation. In total, 122 users were interviewed. Interviews were audio-recorded and transcribed verbatim. Emergent themes were mapped to the Extended Technology Acceptance Model (e-TAM).

RESULTS

Initial perceptions were driven by unfamiliarity with the system. As users became more proficient and efficient in using the CPOE system, additional safety benefits become apparent. However, accompanying increased reports of benefits were reports of usability problems and new types of errors arising from CPOE use. Reports of workarounds appeared for the first time at 6-month interviews. These workarounds were adopted to allow routine work to continue and to attenuate some of the perceived negative consequences of CPOE, including delayed medications and reduced patient interaction.

CONCLUSION

This study is one of the first to examine changes in perceptions of CPOE at multiple points, demonstrating the trajectory of changes in views over time. It provides new information about the time point at which workarounds begin to be embedded in practice and are potentially most receptive to identification and remediation. It suggests an adaptive implementation and support program would be beneficial, as reported difficulties and concerns change during the first six months of use.

Impact of an inpatient electronic prescribing system on prescribing error causation: a qualitative evaluation in an English hospital

BACKGROUND

Few studies have applied a systems approach to understanding the causes of specific prescribing errors in the context of hospital electronic prescribing (EP). A comprehensive understanding of underlying causes is essential for developing effective interventions to improve prescribing safety. Our objectives were to explore prescribers' perspectives of the causes of errors occurring with EP and to make recommendations to maximise benefits and minimise risks.

METHODS

We studied a large hospital using inpatient EP. From April to June 2016, semistructured interviews were conducted with purposively sampled prescribers involved with a prescribing error. Interviews explored prescribers' perceived causes of the error and views about EP; they were audio-recorded and transcribed verbatim. Data were thematically analysed against a framework based on Reason's accident causation model, with a focus on identifying latent conditions.

RESULTS

Twenty-five interviews explored causes of 32 errors. Slips and rule-based mistakes were the most common active failures. Error causation was multifactorial; environmental, individual, team, task and technology error-producing conditions were all influenced by EP. There were three broad groups of latent conditions: the EP system's functionality and design; the organisation's decisions around EP implementation and use; and prescribing behaviours in the context of EP.

CONCLUSIONS

Errors were associated with the design of EP itself and its integration within the healthcare environment. Findings suggest that EP vendors should focus on revolutionising interface design and usability issues, bearing in mind the wider healthcare context in which such software is used. Healthcare organisations should draw upon human factors principles when implementing EP. Consideration of work environment, infrastructure, training, prescribing responsibilities and behaviours should be considered to address local issues identified.

Personalization and Patient Involvement in Decision Support Systems: Current Trends

OBJECTIVES

This survey aims at highlighting the latest trends (2012-2014) on the development, use, and evaluation of Information and Communication Technologies (ICT) based decision support systems (DSSs) in medicine, with a particular focus on patient-centered and personalized care.

METHODS

We considered papers published on scientific journals, by querying PubMed and Web of ScienceTM. Included studies focused on the implementation or evaluation of ICT-based tools used in clinical practice. A separate search was performed on computerized physician order entry systems (CPOEs), since they are increasingly embedding patient-tailored decision support.

RESULTS

We found 73 papers on DSSs (53 on specific ICT tools) and 72 papers on CPOEs. Although decision support through the delivery of recommendations is frequent (28/53 papers), our review highlighted also DSSs only based on efficient information presentation (25/53). Patient participation in making decisions is still limited (9/53), and mostly focused on risk communication. The most represented medical area is cancer (12%). Policy makers are beginning to be included among stakeholders (6/73), but integration with hospital information systems is still low. Concerning knowledge representation/management issues, we identified a trend towards building inference engines on top of standard data models. Most of the tools (57%) underwent a formal assessment study, even if half of them aimed at evaluating usability and not effectiveness.

CONCLUSIONS

Overall, we have noticed interesting evolutions of medical DSSs to improve communication with the patient, consider the economic and organizational impact, and use standard models for knowledge representation. However, systems focusing on patient-centered care still do not seem to be available at large.

An appraisal of published usability evaluations of electronic health records via systematic review

OBJECTIVE

In this systematic review, we aimed to evaluate methodological and reporting trends present in the current literature by investigating published usability studies of electronic health records (EHRs).

METHODS

A literature search was conducted for articles published through January 2015 using MEDLINE (Ovid), EMBASE, Scopus, and Web of Science, supplemented by citation and reference list reviews. Studies were included if they tested the usability of hospital and clinic EHR systems in the inpatient, outpatient, emergency department, or operating room setting.

RESULTS

A total of 4848 references were identified for title and abstract screening. Full text screening was performed for 197 articles, with 120 meeting the criteria for study inclusion.

CONCLUSION

A review of the literature demonstrates a paucity of quality published studies describing scientifically valid and reproducible usability evaluations at various stages of EHR system development. A lack of formal and standardized reporting of EHR usability evaluation results is a major contributor to this knowledge gap, and efforts to improve this deficiency will be one step of moving the field of usability engineering forward.

A Survey of the Literature on Unintended Consequences Associated with Health Information Technology: 2014-2015

OBJECTIVE

To summarize recent research on unintended consequences associated with implementation and use of health information technology (health IT). Included in the review are original empirical investigations published in English between 2014 and 2015 that reported unintended effects introduced by adoption of digital interventions. Our analysis focuses on the trends of this steam of research, areas in which unintended consequences have continued to be reported, and common themes that emerge from the findings of these studies.

METHOD

Most of the papers reviewed were retrieved by searching three literature databases: MEDLINE, Embase, and CINAHL. Two rounds of searches were performed: the first round used more restrictive search terms specific to unintended consequences; the second round lifted the restrictions to include more generic health IT evaluation studies. Each paper was independently screened by at least two authors; differences were resolved through consensus development.

RESULTS

The literature search identified 1,538 papers that were potentially relevant; 34 were deemed meeting our inclusion criteria after screening. Studies described in these 34 papers took place in a wide variety of care areas from emergency departments to ophthalmology clinics. Some papers reflected several previously unreported unintended consequences, such as staff attrition and patients' withholding of information due to privacy and security concerns. A majority of these studies (71%) were quantitative investigations based on analysis of objectively recorded data. Several of them employed longitudinal or time series designs to distinguish between unintended consequences that had only transient impact, versus those that had persisting impact. Most of these unintended consequences resulted in adverse outcomes, even though instances of beneficial impact were also noted. While care areas covered were heterogeneous, over half of the studies were conducted at academic medical centers or teaching hospitals.

CONCLUSION

Recent studies published in the past two years represent significant advancement of unintended consequences research by seeking to include more types of health IT applications and to quantify the impact using objectively recorded data and longitudinal or time series designs. However, more mixed-methods studies are needed to develop deeper insights into the observed unintended adverse outcomes, including their root causes and remedies. We also encourage future research to go beyond the paradigm of simply describing unintended consequences, and to develop and test solutions that can prevent or minimize their impact.

Sustained User Engagement in Health Information Technology: The Long Road from Implementation to System Optimization of Computerized Physician Order Entry and Clinical Decision Support Systems for Prescribing in Hospitals in England

OBJECTIVE

To explore and understand approaches to user engagement through investigating the range of ways in which health care workers and organizations accommodated the introduction of computerized physician order entry (CPOE) and computerized decision support (CDS) for hospital prescribing.

STUDY SETTING

Six hospitals in England, United Kingdom.

STUDY DESIGN

Qualitative case study.

DATA COLLECTION

We undertook qualitative semi-structured interviews, non-participant observations of meetings and system use, and collected organizational documents over three time periods from six hospitals. Thematic analysis was initially undertaken within individual cases, followed by cross-case comparisons.

FINDINGS

We conducted 173 interviews, conducted 24 observations, and collected 17 documents between 2011 and 2015. We found that perceived individual and safety benefits among different user groups tended to facilitate engagement in some, while other less engaged groups developed resistance and unsanctioned workarounds if systems were perceived to be inadequate. We identified both the opportunity and need for sustained engagement across user groups around system enhancement (e.g., through customizing software) and the development of user competencies and effective use.

CONCLUSIONS

There is an urgent need to move away from an episodic view of engagement focused on the preimplementation phase, to more continuous holistic attempts to engage with and respond to end-users.

Perceptions and experiences of the implementation, management, use and optimisation of electronic prescribing systems in hospital settings: protocol for a systematic review of qualitative studies

INTRODUCTION

There is increasing evidence that electronic prescribing (ePrescribing) or computerised provider/physician order entry (CPOE) systems can improve the quality and safety of healthcare services. However, it has also become clear that their implementation is not straightforward and may create unintended or undesired consequences once in use. In this context, qualitative approaches have been particularly useful and their interpretative synthesis could make an important and timely contribution to the field. This review will aim to identify, appraise and synthesise qualitative studies on ePrescribing/CPOE in hospital settings, with or without clinical decision support.

METHODS AND ANALYSIS

Data sources will include the following bibliographic databases: MEDLINE, MEDLINE In Process, EMBASE, PsycINFO, Social Policy and Practice via Ovid, CINAHL via EBSCO, The Cochrane Library (CDSR, DARE and CENTRAL databases), Nursing and Allied Health Sources, Applied Social Sciences Index and Abstracts via ProQuest and SCOPUS. In addition, other sources will be searched for ongoing studies (ClinicalTrials.gov) and grey literature: Healthcare Management Information Consortium, Conference Proceedings Citation Index (Web of Science) and Sociological abstracts. Studies will be independently screened for eligibility by 2 reviewers. Qualitative studies, either standalone or in the context of mixed-methods designs, reporting the perspectives of any actors involved in the implementation, management and use of ePrescribing/CPOE systems in hospital-based care settings will be included. Data extraction will be conducted by 2 reviewers using a piloted form. Quality appraisal will be based on criteria from the Critical Appraisal Skills Programme checklist and Standards for Reporting Qualitative Research. Studies will not be excluded based on quality assessment. A postsynthesis sensitivity analysis will be undertaken. Data analysis will follow the thematic synthesis method.

ETHICS AND DISSEMINATION

The study does not require ethical approval as primary data will not be collected. The results of the study will be published in a peer-reviewed journal and presented at relevant conferences.

TRIAL REGISTRATION NUMBER

CRD42016035552.

Workarounds to hospital electronic prescribing systems: a qualitative study in English hospitals

BACKGROUND

Concerns with the usability of electronic prescribing (ePrescribing) systems can lead to the development of workarounds by users.

OBJECTIVES

To investigate the types of workarounds users employed, the underlying reasons offered and implications for care provision and patient safety.

METHODS

We collected a large qualitative data set, comprising interviews, observations and project documents, as part of an evaluation of ePrescribing systems in five English hospitals, which we conceptualised as case studies. Data were collected at up to three different time points throughout implementation and adoption. Thematic analysis involving deductive and inductive approaches was facilitated by NVivo 10.

RESULTS

Our data set consisted of 173 interviews, 24 rounds of observation and 17 documents. Participating hospitals were at various stages of implementing a range of systems with differing functionalities. We identified two types of workarounds: informal and formal. The former were informal practices employed by users not approved by management, which were introduced because of perceived changes to professional roles, issues with system usability and performance and challenges relating to the inaccessibility of hardware. The latter were formalised practices that were promoted by management and occurred when systems posed threats to patient safety and organisational functioning. Both types of workarounds involved using paper and other software systems as intermediaries, which often created new risks relating to a lack of efficient transfer of real-time information between different users.

CONCLUSIONS

Assessing formal and informal workarounds employed by users should be part of routine organisational implementation strategies of major health information technology initiatives. Workarounds can create new risks and present new opportunities for improvement in system design and integration.

The impact of electronic prescribing systems on pharmacists' time and workflow: protocol for a time-and-motion study in English NHS hospitals

INTRODUCTION

Electronic prescribing (ePrescribing) systems are rapidly being introduced into National Health Systems (NHS) hospitals in England following their widespread earlier adoption into primary care settings. Such systems require substantial changes in the way pharmacists organise their work and perform their roles. There is however as yet limited evidence on the extent to which these changes may support or compromise efficient and safe working practices by pharmacists. Identifying and quantifying these changes, and their effects, is central to informing system and work practice design, as well as informing training and implementation processes. This protocol describes a study to measure the impact of ePrescribing systems on pharmacists' time and workflow.

METHODS AND ANALYSIS

A direct observational controlled pre-post implementation time-and-motion study will be conducted in six wards at one NHS Trust over two observational periods. Pharmacists will be shadowed and details of all work tasks performed will be collected and time-stamped. Task distribution, frequency and duration will be measured and changes in these measures preimplementation and postimplementation, and between control and intervention wards will be measured. Interviews with pharmacists will investigate their perceptions of the impact of the ePrescribing systems on their work and will be conducted in both periods. The extent to which pharmacists' expectations of the impact of the ePrescribing systems on their work with postimplementation reports will be qualitatively explored, as will any differences between perceptions and results from the time-and-motion analysis.

ETHICS AND DISSEMINATION

Institutional research ethics approval has been obtained from The University of Edinburgh. Local approval from the participating NHS Trust and informed consent from participating pharmacists have been obtained, while also complying with local governance requirements. The results of the study will be presented at conferences, published in peer-reviewed journals, and shared with members of our Patient and Public Involvement Group, to facilitate wider dissemination.

Qualitative analysis of vendor discussions on the procurement of Computerised Physician Order Entry and Clinical Decision Support systems in hospitals

OBJECTIVES

We studied vendor perspectives about potentially transferable lessons for implementing organisations and national strategies surrounding the procurement of Computerised Physician Order Entry (CPOE)/Clinical Decision Support (CDS) systems in English hospitals.

SETTING

Data were collected from digitally audio-recorded discussions from a series of CPOE/CDS vendor round-table discussions held in September 2014 in the UK.

PARTICIPANTS

Nine participants, representing 6 key vendors operating in the UK, attended. The discussions were transcribed verbatim and thematically analysed.

RESULTS

Vendors reported a range of challenges surrounding the procurement and contracting processes of CPOE/CDS systems, including hospitals' inability to adequately assess their own needs and then select a suitable product, rushed procurement and implementation processes that resulted in difficulties in meaningfully engaging with vendors, as well as challenges relating to contracting leading to ambiguities in implementation roles. Consequently, relationships between system vendors and hospitals were often strained, the vendors attributing this to a lack of hospital management's appreciation of the complexities associated with implementation efforts. Future anticipated challenges included issues surrounding the standardisation of data to enable their aggregation across systems for effective secondary uses, and implementation of data exchange with providers outside the hospital.

CONCLUSIONS

Our results indicate that there are significant issues surrounding capacity to procure and optimise CPOE/CDS systems among UK hospitals. There is an urgent need to encourage more synergistic and collaborative working between providers and vendors and for a more centralised support for National Health Service hospitals, which draws on a wider body of experience, including a formalised procurement framework with value-based product specifications.

Delivering digital health and well-being at scale: lessons learned during the implementation of the dallas program in the United Kingdom

Objective To identify implementation lessons from the United Kingdom Delivering Assisted Living Lifestyles at Scale (dallas) program—a large-scale, national technology program that aims to deliver a broad range of digital services and products to the public to promote health and well-being.

Materials and Methods Prospective, longitudinal qualitative research study investigating implementation processes. Qualitative data collected includes semi-structured e-Health Implementation Toolkit–led interviews at baseline/mid-point ( n = 38), quarterly evaluation, quarterly technical and barrier and solutions reports, observational logs, quarterly evaluation alignment interviews with project leads, observational data collected during meetings, and ethnographic data from dallas events ( n > 200 distinct pieces of qualitative data). Data analysis was guided by Normalization Process Theory, a sociological theory that aids conceptualization of implementation issues in complex healthcare settings.

Results Five key challenges were identified: 1) The challenge of establishing and maintaining large heterogeneous, multi-agency partnerships to deliver new models of healthcare; 2) The need for resilience in the face of barriers and set-backs including the backdrop of continually changing external environments; 3) The inherent tension between embracing innovative co-design and achieving delivery at pace and at scale ; 4) The effects of branding and marketing issues in consumer healthcare settings; and 5) The challenge of interoperability and information governance, when commercial proprietary models are dominant.

Conclusions The magnitude and ambition of the dallas program provides a unique opportunity to investigate the macro level implementation challenges faced when designing and delivering digital health and wellness services at scale. Flexibility, adaptability, and resilience are key implementation facilitators when shifting to new digitally enabled models of care.