Anatomy of a failure: A sociotechnical evaluation of a laboratory physician order entry system implementation

Exploring the role of professional identity in the implementation of clinical decision support systems-a narrative review

BACKGROUND

Clinical decision support systems (CDSSs) have the potential to improve quality of care, patient safety, and efficiency because of their ability to perform medical tasks in a more data-driven, evidence-based, and semi-autonomous way. However, CDSSs may also affect the professional identity of health professionals. Some professionals might experience these systems as a threat to their professional identity, as CDSSs could partially substitute clinical competencies, autonomy, or control over the care process. Other professionals may experience an empowerment of the role in the medical system. The purpose of this study is to uncover the role of professional identity in CDSS implementation and to identify core human, technological, and organizational factors that may determine the effect of CDSSs on professional identity.

METHODS

We conducted a systematic literature review and included peer-reviewed empirical studies from two electronic databases (PubMed, Web of Science) that reported on key factors to CDSS implementation and were published between 2010 and 2023. Our explorative, inductive thematic analysis assessed the antecedents of professional identity-related mechanisms from the perspective of different health care professionals (i.e., physicians, residents, nurse practitioners, pharmacists).

RESULTS

One hundred thirty-one qualitative, quantitative, or mixed-method studies from over 60 journals were included in this review. The thematic analysis found three dimensions of professional identity-related mechanisms that influence CDSS implementation success: perceived threat or enhancement of professional control and autonomy, perceived threat or enhancement of professional skills and expertise, and perceived loss or gain of control over patient relationships. At the technological level, the most common issues were the system's ability to fit into existing clinical workflows and organizational structures, and its ability to meet user needs. At the organizational level, time pressure and tension, as well as internal communication and involvement of end users were most frequently reported. At the human level, individual attitudes and emotional responses, as well as familiarity with the system, most often influenced the CDSS implementation. Our results show that professional identity-related mechanisms are driven by these factors and influence CDSS implementation success. The perception of the change of professional identity is influenced by the user's professional status and expertise and is improved over the course of implementation.

CONCLUSION

This review highlights the need for health care managers to evaluate perceived professional identity threats to health care professionals across all implementation phases when introducing a CDSS and to consider their varying manifestations among different health care professionals. Moreover, it highlights the importance of innovation and change management approaches, such as involving health professionals in the design and implementation process to mitigate threat perceptions. We provide future areas of research for the evaluation of the professional identity construct within health care.

Evaluating a Prototype Clinical Decision Support Tool for Chronic Pain Treatment in Primary Care

OBJECTIVES

 The Chronic Pain Treatment Tracker (Tx Tracker) is a prototype decision support tool to aid primary care clinicians when caring for patients with chronic noncancer pain. This study evaluated clinicians' perceived utility of Tx Tracker in meeting information needs and identifying treatment options, and preferences for visual design.

METHODS

 We conducted 12 semi-structured interviews with primary care clinicians from four health systems in Indiana. The interviews were conducted in two waves, with prototype and interview guide revisions after the first six interviews. The interviews included exploration of Tx Tracker using a think-aloud approach and a clinical scenario. Clinicians were presented with a patient scenario and asked to use Tx Tracker to make a treatment recommendation. Last, participants answered several evaluation questions. Detailed field notes were collected, coded, and thematically analyzed by four analysts.

RESULTS

 We identified several themes: the need for clinicians to be presented with a comprehensive patient history, the usefulness of Tx Tracker in patient discussions about treatment planning, potential usefulness of Tx Tracker for patients with high uncertainty or risk, potential usefulness of Tx Tracker in aggregating scattered information, variability in expectations about workflows, skepticism about underlying electronic health record data quality, interest in using Tx Tracker to annotate or update information, interest in using Tx Tracker to translate information to clinical action, desire for interface with visual cues for risks, warnings, or treatment options, and desire for interactive functionality.

CONCLUSION

 Tools like Tx Tracker, by aggregating key information about past, current, and potential future treatments, may help clinicians collaborate with their patients in choosing the best pain treatments. Still, the use and usefulness of Tx Tracker likely relies on continued improvement of its functionality, accurate and complete underlying data, and tailored integration with varying workflows, care team roles, and user preferences.

Contrast of Digital and Health Literacy Between IT and Health Care Specialists Highlights the Importance of Multidisciplinary Teams for Digital Health-A Pilot Study

PURPOSE

Despite their promises, digital innovations have scarcely translated to technologies used in routine clinical practice, making the identification of barriers to successful implementation a research priority. Low levels of transdisciplinary skills represent such a barrier but so far, this has not been evaluated and compared between information technology (IT) and health care specialists. In this study, we evaluated the level of digital health literacy among IT and health care specialists.

MATERIALS AND METHODS

An anonymous questionnaire was distributed to staff at a breast cancer unit and an IT department of two German universities in December 2020. The survey questionnaire consisted of the previously validated eHealth Literacy Assessment Toolkit and additional questions with respect to age, profession, and career stage. Mann-Whitney or Wilcoxon rank-sum tests and two-sample chi-square tests were used for the analysis.

RESULTS

The survey was completed by 113 individuals: 70 (61.9%) IT specialists and 43 (38.1%) health care specialists. Health care specialists scored significantly higher on the health-related scales and IT specialists scored significantly higher on the digitally related scales. No single participant identified themselves to have the highest level of literacy on all survey questions (n = 0 of 113; 0%). Only one person (n = 1 of 113; 0.9%) consistently reported a high or the highest level of literacy.

CONCLUSION

Although IT and health care specialists showed great literacy in their respective disciplines, only few individuals combined both digital and health care literacy. Multidisciplinary teams and transdisciplinary curricula are crucial to bridge skill gaps between disciplines and to drive the implementation of digital health initiatives.

Development and evaluation of an electronic hospital referral system: a human factors approach

Coordinating care across hospitals has been identified as a patient safety risk as referrals are often paper-based and poorly documented. Electronic referral systems have the potential to improve the situation but can fail to gain uptake. We applied a human factors/ergonomics (HFE) approach to place analysis of local workflow and user engagement central to the development of a new regional electronic referral system. The intervention was evaluated with a before-and-after study. Referral quality improved, referrals containing sufficient clinical information for continuation of care increased from 36.9% to 83.5% and completeness of referral information significantly improved. There was a 35.7% reduction in the number of calls to the on-call specialist, and the mean period between admission and surgery for expedited transfers was reduced. Applying HFE informed design with use-based evidence; the system maintains sustained uptake three years after implementation. Reliable recording of information translates to better patient safety during inter-hospital transitions. Practitioners summary: This study developed, implemented and evaluated a clinical referral system using a human factors approach. Process analysis and usability studies were used to inform the application requirements and design. Region-wide implementation in hospitals resulted in the improved quality and completeness of clinical referral information and efficiencies in the referral process.

The Role of Frontline Leaders in Building Health Professional Support for a New Patient Portal: Survey Study

Background

Effective leadership and change management are thought to contribute to the successful implementation of health information technology innovations. However, limited attention has been paid to the role of frontline leaders in building health professional support for new technical innovations.

Objective

First, we examined whether frontline leaders’ positive expectations of a patient portal and perceptions of its implementation were associated with their support for the portal. Second, we explored whether leaders’ positive perceptions influenced the same unit’s health professional support for the portal.

Methods

Data were collected through an online survey of 2067 health professionals and 401 frontline leaders working in 44 units from 14 health organizations in Finland. The participating organizations run a joint self-care and digital value services project developing a new patient portal for self-management. The survey was conducted before the piloting and implementation of the patient portal.

Results

The frontline leaders’ perception of vision clarity had the strongest association with their own support for the portal (ß=.40, P<.001). Results also showed an association between leaders’ view of organizational readiness and their support (ß=.15, P=.04). The leaders’ positive perceptions of the quality of informing about the patient portal was associated with both leaders’ own (ß=.16, P=.02) and subordinate health professionals’ support for the portal (ß=.08, P<.001). Furthermore, professional participation in the planning of the portal was positively associated with their support (ß=.57, P<.001).

Conclusions

Findings suggest that assuring good informing, communicating a clear vision to frontline leaders, and acknowledging organizational readiness for change can increase health professional support for electronic health (eHealth) services in the pre-implementation phase. Results highlight the role of frontline leaders in engaging professionals in the planning and implementation of eHealth services and in building health professionals’ positive attitudes toward the implementation of eHealth services.

[Online information for relatives of critically ill patients : Pilot test of the usability of an ICU families website]

BACKGROUND

Relatives of intensive care patients have a very high need for information. This is due to the acute and serious, often life-threatening illness of the patients and the very complex and technical environment of an intensive care unit (ICU). Unmet needs for information can increase anxiety, sleep disorders, stress, and depressive symptoms in the relatives.

OBJECTIVES

The potential of the ICU families website in terms of usability and functionality during real-time testing were evaluated.

METHODS

The ICU families project created a dynamic online information platform in the form of a password-protected website. It contains pictures, written explanations, 5 movies, a forum and a diary function. The usability of the website was tested among 10 lay people and 10 experts (7 nurses and 3 physicians) according to the Think Aloud Method.

RESULTS

The outcome is qualitative feedback based on video documentation by laypeople and suggestions by experts. Criticisms mentioned by the test subjects were insufficient image material, small size of the operator contact link and lack of a home button. With a mean of 9.1 (rating scale, 0 = very poor, 10 = very good), the website was almost universally recommended by the experts.

CONCLUSIONS

This usability test of a website for relatives of ICU patients conducted among 20 test subjects showed the biggest challenges related to solving individual test scenarios and provided valuable hints for improving website usability. Features of the website highlighted as positive were the clear layout, the symbols, the diary and the consideration of children. This information was used to improve the site for subsequent roll-out in a randomized, controlled and multicentre study.

Health professionals' expectations of a national patient portal for self-management

OBJECTIVE

Patient portals have the potential to support patient empowerment, self-care, and management, but their adoption and use have reported to be limited. Patients' more active role creates tension, as health professionals need to change their traditional expert role and share control with patients. Professionals may also have other expectations and concerns that influence the acceptance of patient portals supporting patient empowerment. This study explores the health professionals' expectations influencing their support for a new patient portal for self-management prior to implementation.

DESIGN

The study empirically evaluates the impact of several variables on health professionals' support for a new patient portal for self-management. The study variables include 1) expected influences on professionals' work, 2) expected influences on patients, 3) usability, 4) professional autonomy, 5) informing, 6) implementation practices, and 7) user participation.

METHODS

Data was collected through an online survey of 2943 health professionals working in 14 health organizations in Finland. The participating organizations run a joint Self-Care and Digital Value Services (ODA) project, developing a national patient portal for self-management. Three main services of the patient portal are well-being coaching, diagnostic tool, and a health care plan.

RESULTS AND CONCLUSIONS

The results show that health professionals' positive expectations about the new patient portal, adequate informing of professionals ahead of time, and the organization's good implementation practices had a positive impact on their support for the patient portal. Perceived threat to professional autonomy had a negative impact on professionals' support for the portal. Age, gender, and user participation did not influence support. Professionals' concerns were related especially to patients' willingness and capability to use the patient portal. The findings can guide health care providers to facilitate professionals' support and remove obstacles to introduce patient portals already in the pre-implementation phase.

A Systematic Investigation on Barriers and Critical Success Factors for Clinical Information Systems in Integrated Care Settings

OBJECTIVES

Clinical Information Systems (CIS) have ever since the introduction of information technology in healthcare played an important role to support healthcare professionals and the process of treatment. With the rise of the concept of integrated care organizational borders, the sole focus on data aggregation or healthcare professionals as users disappear more and more. The manuscript discusses the concept of CISs and investigates critical success factors for CISs in the context of integrated care and in the course of time.

METHODS

In order to identify critical success factors and barriers for CISs a systematic literature review was conducted based on the results from PubMed and Cochrane, using MaxQDA. Search results were thereby limited to reviews or meta-analysis.

RESULTS

We have found 1919 references of which 40 met the inclusion criteria. The analysis of the manuscripts resulted in a comprehensive list of success factors and barriers related to CISs in integrated care settings. Most barriers were user-related whereas for the success factors an even distribution of organizational, technical and user-related factors was observed. The vast majority of publications was focused on healthcare professionals.

CONCLUSION

It is important to incorporate experiences made/ collected over time, as the problems encountered seem to remain almost unvaried. In order to support further systematic investigations on the topic it is necessary to rethink existing concepts and definitions to realign them with the ideas of integrated care.

Physician satisfaction with transition from CPOE to paper-based prescription

INTRODUCTION

In January 2015, Rouen University Hospital's information system experienced serious issues. It was necessary to rapidly switch from the computerized provider order entry (CPOE) system towards a paper-based order entry (PBOE) system. This was an opportunity to evaluate prescriber opinion on the two provider order entry (POE) systems.

METHODS

All residents were asked to fill an augmented version of the POE satisfaction and usage survey for both POE systems. The results were compared to identify the strengths and weaknesses of each system.

RESULTS

Fifty-one respondents had used the CPOE system and the PBOE system. Overall, satisfaction was higher with PBOE than CPOE (odds ratio (OR)=3.74; p<0.001). Usability (OR=4.00; p<0.001), reliability (OR=8.54; p<0.001), time consumption (OR=0.50; p<0.05 - survey statement was formulated negatively), and communication with nurses (OR=14.27; p<0.0001) reached statistically better agreement. The more experience with CPOE the more residents were disillusioned with the reliability (OR=6.55; p<0.01), the usability (OR=5.68; p<0.01) and the patient safety (OR=0.27; p<0.05 - survey statement was formulated negatively) of CPOE. Although safety issues were reported for both systems, the causes were different; PBOE imposed frequent rewriting of the order while CPOE lack of usability might be unsafe. Another important issue with both POE systems was time consumption.

CONCLUSION

Residents did not report any increase in safety issues with the rapid switch from CPOE to PBOE. They even seemed more satisfied with the rollback to paper, which remains a possible degraded mode in case of health information technology collapse.

Cognitive engineering and health informatics: Applications and intersections

Cognitive engineering is an applied field with roots in both cognitive science and engineering that has been used to support design of information displays, decision support, human-automation interaction, and training in numerous high risk domains ranging from nuclear power plant control to transportation and defense systems. Cognitive engineering provides a set of structured, analytic methods for data collection and analysis that intersect with and complement methods of Cognitive Informatics. These methods support discovery of aspects of the work that make performance challenging, as well as the knowledge, skills, and strategies that experts use to meet those challenges. Importantly, cognitive engineering methods provide novel representations that highlight the inherent complexities of the work domain and traceable links between the results of cognitive analyses and actionable design requirements. This article provides an overview of relevant cognitive engineering methods, and illustrates how they have been applied to the design of health information technology (HIT) systems. Additionally, although cognitive engineering methods have been applied in the design of user-centered informatics systems, methods drawn from informatics are not typically incorporated into a cognitive engineering analysis. This article presents a discussion regarding ways in which data-rich methods can inform cognitive engineering.

Mind the Gap. A systematic review to identify usability and safety challenges and practices during electronic health record implementation

OBJECTIVE

Decisions made during electronic health record (EHR) implementations profoundly affect usability and safety. This study aims to identify gaps between the current literature and key stakeholders' perceptions of usability and safety practices and the challenges encountered during the implementation of EHRs.

MATERIALS AND METHODS

Two approaches were used: a literature review and interviews with key stakeholders. We performed a systematic review of the literature to identify usability and safety challenges and best practices during implementation. A total of 55 articles were reviewed through searches of PubMed, Web of Science and Scopus. We used a qualitative approach to identify key stakeholders' perceptions; semi-structured interviews were conducted with a diverse set of health IT stakeholders to understand their current practices and challenges related to usability during implementation. We used a grounded theory approach: data were coded, sorted, and emerging themes were identified. Conclusions from both sources of data were compared to identify areas of misalignment.

RESULTS

We identified six emerging themes from the literature and stakeholder interviews: cost and resources, risk assessment, governance and consensus building, customization, clinical workflow and usability testing, and training. Across these themes, there were misalignments between the literature and stakeholder perspectives, indicating major gaps.

DISCUSSION

Major gaps identified from each of six emerging themes are discussed as critical areas for future research, opportunities for new stakeholder initiatives, and opportunities to better disseminate resources to improve the implementation of EHRs.

CONCLUSION

Our analysis identified practices and challenges across six different emerging themes, illustrated important gaps, and results suggest critical areas for future research and dissemination to improve EHR implementation.

Impact of Patient-centered eHealth Applications on Patient Outcomes: A Review on the Mediating Influence of Human Factor Issues

OBJECTIVES

To examine the evidence of the impact of patient- centered eHealth applications on patient care and to analyze if and how reported human factor issues mediated the outcomes.

METHODS

We searched PubMed (2014-2015) for studies evaluating the impact of patient-centered eHealth applications on patient care (behavior change, self-efficacy, and patient health-related outcomes). The Systems Engineering Initiative for Patient Safety (SEIPS 2.0) model was used as a guidance framework to identify the reported human factors possibly impacting the effectiveness of an eHealth intervention.

RESULTS

Of the 348 potentially relevant papers, 10 papers were included for data analysis. None of the 10 papers reported a negative impact of the eHealth intervention. Seven papers involved a randomized controlled trial (RCT) study. Six of these RCTs reported a positive impact of the eHealth intervention on patient care. All 10 papers reported on human factor issues possibly mediating effects of patient-centered eHealth. Human factors involved patient characteristics, perceived social support, and (type of) interaction between patient and provider.

CONCLUSION

While the amount of patient-centered eHealth interventions increases, many questions remain as to whether and to what extent human factors mediate their use and impact. Future research should adopt a formal theory-driven approach towards human factors when investigating those factors' influence on the effectiveness of these interventions. Insights could then be used to better tailor the content and design of eHealth solutions according to patient user profiles, so as to enhance eHealth interventions impact on patient behavior, self-efficacy, and health-related outcomes.

Prioritizing factors influencing nurses' satisfaction with hospital information systems: a fuzzy analytic hierarchy process approach

Our aim was to use the fuzzy analytic hierarchy process approach to prioritize the factors that influence nurses' satisfaction with a hospital information system. First, we reviewed the related literature to identify and select possible factors. Second, we developed an analytic hierarchy process framework with three main factors (quality of services, of systems, and of information) and 22 subfactors. Third, we developed a questionnaire based on pairwise comparisons and invited 10 experienced nurses who were identified through snowball sampling to rate these factors. Finally, we used Chang's fuzzy extent analysis method to compute the weights of these factors and prioritize them. We found that information quality was the most important factor (58%), followed by service quality (22%) and then system quality (19%). In conclusion, although their weights were not similar, all factors were important and should be considered in evaluating nurses' satisfaction.

Understanding barriers and facilitators to the use of Clinical Information Systems for intensive care units and Anesthesia Record Keeping: A rapid ethnography

OBJECTIVE

This study evaluated the current use of commercial-off-the-shelf Clinical Information Systems (CIS) for intensive care units (ICUs) and Anesthesia Record Keeping (ARK) for operating rooms and post-anesthesia care recovery settings at three Veterans Affairs Medical Centers (VAMCs). Clinicians and administrative staff use these applications at bedside workstations, in operating rooms, at nursing stations, in physician's rooms, and in other various settings. The intention of a CIS or an ARK system is to facilitate creation of electronic records of data, assessments, and procedures from multiple medical devices. The US Department of Veterans Affairs (VA) Office of the Chief of Nursing Informatics sought to understand usage barriers and facilitators to optimize these systems in the future. Therefore, a human factors study was carried out to observe the CIS and ARK systems in use at three VAMCs in order to identify best practices and suggested improvements to currently implemented CIS and ARK systems.

METHODS

We conducted a rapid ethnographic study of clinical end-users interacting with the CIS and ARK systems in the critical care and anesthesia care areas in each of three geographically distributed VAMCs. Two observers recorded interactions and/or interview responses from 88 CIS and ARK end-users. We coded and sorted into logical categories field notes from 69 shadowed participants. The team transcribed and combined data from key informant interviews with 19 additional participants with the observation data. We then integrated findings across observations into meaningful patterns and abstracted the data into themes, which translated directly to barriers to effective adoption and optimization of the CIS and ARK systems.

RESULTS

Effective optimization of the CIS and ARK systems was impeded by: (1) integration issues with other software systems; (2) poor usability; (3) software challenges; (4) hardware challenges; (5) training concerns; (6) unclear roles and lack of coordination among stakeholders; and (7) insufficient technical support. Many of these barriers are multi-faceted and have associated sub-barriers, which are described in detail along with relevant quotes from participants. In addition, regionalized purchases of different CIS and ARK systems, as opposed to enterprise level purchases, contributed to some of the identified barriers. Facilitators to system use included (1) automation and (2) a dedicated facility-level CIS-ARK Coordinator.

CONCLUSIONS

We identified barriers that explain some of the challenges with the optimization of the CIS and ARK commercial systems across the Veterans Health Administration (VHA). To help address these barriers, and evolve them into facilitators, we categorized report findings as (1) interface and system-level changes that vendors or VA healthcare systems can implement; (2) implementation factors under VA control and not under VA control; and (3) factors that may be used to inform future application purchases. We outline several recommendations for improved adoption of CIS and ARK systems and further recommend that human factors engineering and usability requirements become an integral part of VA health information technology (HIT) application procurement, customization, and implementation in order to help eliminate or mitigate some of the barriers of use identified in this study. Human factors engineering methods can be utilized to apply a user-centered approach to application requirements specification, application evaluation, system integration, and application implementation.

Pathologists as Stewards of Laboratory Information

Just as electronic health records are transforming the practice of medicine and health care information management, practicing in the era of the electronic health record offers opportunities, if not imperatives, for pathologists to take on new and “transformative” professional and leadership roles for the organizations they serve. Experience indicates that clinicians will perceive pathologists and laboratories as responsible for all aspects of laboratory testing and information management, including order entry and results reporting, even though such functions may fall beyond the control of the laboratory. As described and expanded upon in the previous 4 articles of this series, the use of electronic health records dictates changes in how clinicians interact with laboratory information. In this environment, pathologists are uniquely positioned to act as the stewards for laboratory information in electronic health records and throughout health care organizations.

Modeling workflow to design machine translation applications for public health practice

OBJECTIVE

Provide a detailed understanding of the information workflow processes related to translating health promotion materials for limited English proficiency individuals in order to inform the design of context-driven machine translation (MT) tools for public health (PH).

MATERIALS AND METHODS

We applied a cognitive work analysis framework to investigate the translation information workflow processes of two large health departments in Washington State. Researchers conducted interviews, performed a task analysis, and validated results with PH professionals to model translation workflow and identify functional requirements for a translation system for PH.

RESULTS

The study resulted in a detailed description of work related to translation of PH materials, an information workflow diagram, and a description of attitudes towards MT technology. We identified a number of themes that hold design implications for incorporating MT in PH translation practice. A PH translation tool prototype was designed based on these findings.

DISCUSSION

This study underscores the importance of understanding the work context and information workflow for which systems will be designed. Based on themes and translation information workflow processes, we identified key design guidelines for incorporating MT into PH translation work. Primary amongst these is that MT should be followed by human review for translations to be of high quality and for the technology to be adopted into practice.

CONCLUSION

The time and costs of creating multilingual health promotion materials are barriers to translation. PH personnel were interested in MT's potential to improve access to low-cost translated PH materials, but expressed concerns about ensuring quality. We outline design considerations and a potential machine translation tool to best fit MT systems into PH practice.

Health information technology and sociotechnical systems: a progress report on recent developments within the UK National Health Service (NHS)

This paper summarises some of the research that Ken Eason and colleagues at Loughborough University have carried out in the last few years on the introduction of Health Information Technologies (HIT) within the UK National Health Service (NHS). In particular, the paper focuses on three examples which illustrate aspects of the introduction of HIT within the NHS and the role played by the UK National Programme for Information Technology (NPfIT). The studies focus on stages of planning and preparation, implementation and use, adaptation and evolution of HIT (e.g., electronic patient records, virtual wards) within primary, secondary and community care settings. Our findings point to a number of common themes which characterise the use of these systems. These include tensions between national and local strategies for implementing HIT and poor fit between healthcare work systems and the design of HIT. The findings are discussed in the light of other large-scale, national attempts to introduce similar technologies, as well as drawing out a set of wider lessons learnt from the NPfIT programme based on Ken Eason's earlier work and other research on the implementation of large-scale HIT.

Determining of factors influencing the success and failure of hospital information system and their evaluation methods: a systematic review

BACKGROUND

Nowadays, using new information technology (IT) has provided remarkable opportunities to decrease medical errors, support health care specialist, increase the efficiency and even the quality of patient's care and safety.

OBJECTIVES

The purpose of this study was the identification of Hospital Information System (HIS) success and failure factors and the evaluation methods of these factors. This research emphasizes the need to a comprehensive evaluation of HISs which considers a wide range of success and failure factors in these systems.

MATERIALS AND METHODS

We searched for relevant English language studies based on keywords in title and abstract, using PubMed, Ovid Medline (by applying MeSH terms), Scopus, ScienceDirect and Embase (earliest entry to march 17, 2012). Studies which considered success models and success or failure factors, or studied the evaluation models of HISs and the related ones were chosen. Since the studies used in this systematic review were heterogeneous, the combination of extracted data was carried out by using narrative synthesis method.

RESULTS

We found 16 articles which required detailed analysis. Finally, the suggested framework includes 12 main factors (functional, organizational, behavioral, cultural, management, technical, strategy, economy, education, legal, ethical and political factors), 67 sub factors, and 33 suggested methods for the evaluation of these sub factors.

CONCLUSIONS

The results of the present research indicates that the emphasis of the HIS evaluation moves from technical subjects to human and organizational subjects, and from objective to subjective issues. Therefore, this issue entails more familiarity with more qualitative evaluation methods. In most of the reviewed studies, the main focus has been laid on the necessity of using multi-method approaches and combining methods to obtain more comprehensive and useful results.

Sources of variation in primary care clinical workflow: implications for the design of cognitive support

This article identifies sources of variation in clinical workflow and implications for the design and implementation of electronic clinical decision support. Sources of variation in workflow were identified via rapid ethnographic observation, focus groups, and interviews across a total of eight medical centers in both the Veterans Health Administration and academic medical centers nationally regarded as leaders in developing and using clinical decision support. Data were reviewed for types of variability within the social and technical subsystems and the external environment as described in the sociotechnical systems theory. Two researchers independently identified examples of variation and their sources, and then met with each other to discuss them until consensus was reached. Sources of variation were categorized as environmental (clinic staffing and clinic pace), social (perception of health information technology and real-time use with patients), or technical (computer access and information access). Examples of sources of variation within each of the categories are described and discussed in terms of impact on clinical workflow. As technologies are implemented, barriers to use become visible over time as users struggle to adapt workflow and work practices to accommodate new technologies. Each source of variability identified has implications for the effective design and implementation of useful health information technology. Accommodating moderate variability in workflow is anticipated to avoid brittle and inflexible workflow designs, while also avoiding unnecessary complexity for implementers and users.

Effects of laboratory data exchange in the care of patients with HIV

BACKGROUND

Electronic health record (EHR) systems are often modified through the addition of new features over time. Few studies have examined the specific effects of such changes. We examined whether implementation of a bidirectional laboratory interface for order entry and data reporting within an existing ambulatory EHR would result in more prompt responses to laboratory indications for antiretroviral therapy (ART) changes or in improved communication with HIV+ patients about relevant laboratory results.

METHODS

We conducted a single-arm intervention study comparing the timeliness of ART regimen changes, HIV viral load (VL) outcomes and patient-reported assessments of care before and after implementation of a laboratory data exchange interface within an existing EHR, without changing the EHR ordering or results reporting user interface. Patient data was extracted from the EHR covering the period from 1 year before to 2 years after the intervention for a cohort of 1181 patients who had received care during the baseline year. The timeliness of ART changes was represented by the days from a laboratory-result "signal" (CD4 dropping below 350 or 200 or VL increasing by a half-log or to a value over 100,000) to an ART-change "response". Patient assessments of care were collected by interviewing 100 anonymous patients at baseline and another 125 at 2 years post-intervention.

RESULTS

A total of 171 laboratory "signal" events were followed within 80 days by a change in ART therapy. The mean time from signal to therapy change (adjusted for clustering by patient) initially increased, from 37.7 days during the pre-intervention year to 48.2 days during the quarter immediately following activation of the lab intervention. It then declined to a mean of 31.4 days over the remaining 21 months of observation (P=0.03 for the 6-day improvement from the pre-period). A majority of patients (65%) achieved undetectable VL values by the end of the observation period; faster signal-response times were not associated with greater achievement of undetectable VL. Patients rated communication about laboratory tests more highly after implementation of the interface (91 vs. 83 on a 100-point scale, P=0.01); ratings were not higher for other aspects of care.

CONCLUSIONS

Adding laboratory data exchange interfaces within existing EHRs holds promise for improving HIV care, both in the timeliness of responses to important laboratory results and in the quality of provider communication about lab tests. However, the benefits from this incremental change may be modest unless more extensive redesign of laboratory follow-up workflows is undertaken, with support from enhanced user interfaces that take advantage of the laboratory information delivered. Providers should also consider increased staffing to compensate for dips in follow-up performance during the initial post-implementation months.

Changes, disruption and innovation: An investigation of the introduction of new health information technology in a microbiology laboratory

Background:

It is expected that health information technology (HIT) will deliver a safer, more efficient and effective health care system. The aim of this study was to undertake a qualitative and video-ethnographic examination of the impact of information technologies on work processes in the reception area of a Microbiology Department, to ascertain what changed, how it changed and the impact of the change.

Materials and Methods:

The setting for this study was the microbiology laboratory of a large tertiary hospital in Sydney. The study consisted of qualitative (interview and focus group) data and observation sessions for the period August 2005 to October 2006 along with video footage shot in three sessions covering the original system and the two stages of the Cerner implementation. Data analysis was assisted by NVivo software and process maps were produced from the video footage.

Results:

There were two laboratory information systems observed in the video footage with computerized provider order entry introduced four months later. Process maps highlighted the large number of pre data entry steps with the original system whilst the newer system incorporated many of these steps in to the data entry stage. However, any time saved with the new system was offset by the requirement to complete some data entry of patient information not previously required. Other changes noted included the change of responsibilities for the reception staff and the physical changes required to accommodate the increased activity around the data entry area.

Conclusions:

Implementing a new HIT is always an exciting time for any environment but ensuring that the implementation goes smoothly and with minimal trouble requires the administrator and their team to plan well in advance for staff training, physical layout and possible staff resource reallocation.

Errors associated with outpatient computerized prescribing systems

OBJECTIVE

To report the frequency, types, and causes of errors associated with outpatient computer-generated prescriptions, and to develop a framework to classify these errors to determine which strategies have greatest potential for preventing them.

MATERIALS AND METHODS

This is a retrospective cohort study of 3850 computer-generated prescriptions received by a commercial outpatient pharmacy chain across three states over 4 weeks in 2008. A clinician panel reviewed the prescriptions using a previously described method to identify and classify medication errors. Primary outcomes were the incidence of medication errors; potential adverse drug events, defined as errors with potential for harm; and rate of prescribing errors by error type and by prescribing system.

RESULTS

Of 3850 prescriptions, 452 (11.7%) contained 466 total errors, of which 163 (35.0%) were considered potential adverse drug events. Error rates varied by computerized prescribing system, from 5.1% to 37.5%. The most common error was omitted information (60.7% of all errors).

DISCUSSION

About one in 10 computer-generated prescriptions included at least one error, of which a third had potential for harm. This is consistent with the literature on manual handwritten prescription error rates. The number, type, and severity of errors varied by computerized prescribing system, suggesting that some systems may be better at preventing errors than others.

CONCLUSIONS

Implementing a computerized prescribing system without comprehensive functionality and processes in place to ensure meaningful system use does not decrease medication errors. The authors offer targeted recommendations on improving computerized prescribing systems to prevent errors.

Patient source of learning about health technologies and ratings of trust in technologies used in their care

In order to design effective health technologies and systems, it is important to understand how patients learn and make decisions about health technologies used in their care. The objective of this study was to examine patients' source of learning about technologies used in their care and how the source related to their trust in the technology was used. Individual face-to-face and telephone interviews were conducted with 24 patients. Altogether, 13 unique sources of information about technology were identified and three major themes emerged: outside of the work system vs. inside the work system; when the health information was provided; the medium used. Patients used multiple sources outside of the healthcare work system to learn about technologies that will be used in their care. Results showed a relationship between learning about technologies from web sources and trust in technologies but no relationship between learning about technologies from healthcare providers and trust in technologies. STATEMENT OF RELEVANCE: The value of considering human attitudes about elements in health systems has been illustrated. This research shows a relationship between patient attitudes about medical technologies used in their care and healthcare work system design. Results show that patient attitudes are formed about technologies used in their care by sources within and outside of the sociotechnical work system.

A Sociotechnical Approach to Evaluating the Impact of ICT on Clinical Care Environments

INTRODUCTION

Process-supporting information technology holds the potential to increase efficiency, reduce errors, and alter professional roles and responsibilities in a manner which allows improvement in the delivery of patient care. However, clashes between the model of health care work inscribed in these tools with the actual nature of work has resulted in staff resistance and decreased organisational uptake of ICT, as well as the facilitation of unexpected and negative effects in efficiency and patient safety. Sociotechnical theory provides a paradigm against which workflow and transfusion of ICT in healthcare could be better explored and understood.

DESIGN

This paper will conceptualise a formative, multi-method longitudinal evaluation process to explore the impact of ICT with an appreciation of the relationship between the social and technical systems within a clinical department.

METHOD

Departmental culture, including clinical work processes and communication patterns will be thoroughly explored before system implementation using both quantitative and qualitative research methods. Findings will be compared with post implementation data, which will incorporate measurement of safety and workflow efficiency indicators.

DISCUSSION

Sociotechnical theory provides a paradigm against which workflow and transfusion of ICT in healthcare could be better explored and understood. However, sociotechnical and multimethod approaches to evaluation do not exist without criticism. Inherent in the protocol are limitations of sociotechnical theory and criticism of the multimethod approach; testing of the methodology in real clinical settings will serve to verify efficacy and refine the process.