Determinants of success of inpatient clinical information systems: a literature review

Nurses' perception of the use of computerised information systems in practice: questionnaire development

AIM

To develop and validate a questionnaire to explore the perceptions of nurses about the implementation of a computerised information system in clinical practice.

BACKGROUND

A growing interest in understanding nurses' experience of developing and implementing clinically relevant Information Technology systems and the lack of measurement tools in this area, justifies further research into the development of instruments to provide an insight into nurses' experience.

DESIGN

Survey and questionnaire development.

METHOD

An initial draft of the questionnaire was developed based on the literature and expert opinion. The questionnaire was piloted by ten nurses to check face validity, reliability and test-retest reliability. A revised version of the questionnaire was distributed to nurses working in the in-patient area of a university hospital in Spain (n = 227). Principal components analysis with oblique rotation was carried out to test theoretically developed underlying dimensions and to test construct validity. Cronbach's alpha coefficient was used to determine internal consistency.

RESULTS

Cronbach's alpha for all the items included in the different scales was 0.88 in the pilot questionnaire and test-retest reliability was adequate. Principal components analysis of items related to mechanisms produced a three-component structure ('IT support', 'usability' and 'information characteristics'). The three factors explained 48.6% of the total variance and Cronbach's alpha ranged from 0.66-0.79. Principal components analysis of items related to outcomes produced a three factor solution ('impact on patient care', 'impact on communication' and 'image profile'). The factors explained 65.9% of the total variance and Cronbach's alpha ranged from 0.64-0.85.

CONCLUSION

The study provides a detailed description and justification of an instrument development process. The instrument is valid and reliable for the setting where it has been used.

RELEVANCE TO CLINICAL PRACTICE

The instrument could provide insight into nurses' experience of IT implementation that will guide further development of systems to enhance clinical practice.

Do electronic health records affect the patient-psychiatrist relationship? A before & after study of psychiatric outpatients

BACKGROUND

A growing body of literature shows that patients accept the use of computers in clinical care. Nonetheless, studies have shown that computers unequivocally change both verbal and non-verbal communication style and increase patients' concerns about the privacy of their records. We found no studies which evaluated the use of Electronic Health Records (EHRs) specifically on psychiatric patient satisfaction, nor any that took place exclusively in a psychiatric treatment setting. Due to the special reliance on communication for psychiatric diagnosis and evaluation, and the emphasis on confidentiality of psychiatric records, the results of previous studies may not apply equally to psychiatric patients.

METHOD

We examined the association between EHR use and changes to the patient-psychiatrist relationship. A patient satisfaction survey was administered to psychiatric patient volunteers prior to and following implementation of an EHR. All subjects were adult outpatients with chronic mental illness.

RESULTS

Survey responses were grouped into categories of "Overall," "Technical," "Interpersonal," "Communication & Education,," "Time," "Confidentiality," "Anxiety," and "Computer Use." Multiple, unpaired, two-tailed t-tests comparing pre- and post-implementation groups showed no significant differences (at the 0.05 level) to any questionnaire category for all subjects combined or when subjects were stratified by primary diagnosis category.

CONCLUSIONS

While many barriers to the adoption of electronic health records do exist, concerns about disruption to the patient-psychiatrist relationship need not be a prominent focus. Attention to communication style, interpersonal manner, and computer proficiency may help maintain the quality of the patient-psychiatrist relationship following EHR implementation.

Organization-wide adoption of computerized provider order entry systems: a study based on diffusion of innovations theory

Background

Computerized provider order entry (CPOE) systems have been introduced to reduce medication errors, increase safety, improve work-flow efficiency, and increase medical service quality at the moment of prescription. Making the impact of CPOE systems more observable may facilitate their adoption by users. We set out to examine factors associated with the adoption of a CPOE system for inter-organizational and intra-organizational care.

Methods

The diffusion of innovation theory was used to understand physicians' and nurses' attitudes and thoughts about implementation and use of the CPOE system. Two online survey questionnaires were distributed to all physicians and nurses using a CPOE system in county-wide healthcare organizations. The number of complete questionnaires analyzed was 134 from 200 nurses (67.0%) and 176 from 741 physicians (23.8%). Data were analyzed using descriptive-analytical statistical methods.

Results

More nurses (56.7%) than physicians (31.3%) stated that the CPOE system introduction had worked well in their clinical setting (P < 0.001). Similarly, more physicians (73.9%) than nurses (50.7%) reported that they found the system not adapted to their specific professional practice (P = < 0.001). Also more physicians (25.0%) than nurses (13.4%) stated that they did want to return to the previous system (P = 0.041). We found that in particular the received relative advantages of the CPOE system were estimated to be significantly (P < 0.001) higher among nurses (39.6%) than physicians (16.5%). However, physicians' agreements with the compatibility of the CPOE and with its complexity were significantly higher than the nurses (P < 0.001).

Conclusions

Qualifications for CPOE adoption as defined by three attributes of diffusion of innovation theory were not satisfied in the study setting. CPOE systems are introduced as a response to the present limitations in paper-based systems. In consequence, user expectations are often high on their relative advantages as well as on a low level of complexity. Building CPOE systems therefore requires designs that can provide rather important additional advantages, e.g. by preventing prescription errors and ultimately improving patient safety and safety of clinical work. The decision-making process leading to the implementation and use of CPOE systems in healthcare therefore has to be improved. As any change in health service settings usually faces resistance, we emphasize that CPOE system designers and healthcare decision-makers should continually collect users' feedback about the systems, while not forgetting that it also is necessary to inform the users about the potential benefits involved.

Health information system implementation: a qualitative meta-analysis

Healthcare information systems (HISs) are often implemented to enhance the quality of care and the degree to which it is patient-centered, as well as to improve the efficiency and safety of services. However, the outcomes of HIS implementations have not met expectations. We set out to organize the knowledge gained in qualitative studies performed in association with HIS implementations and to use this knowledge to outline an updated structure for implementation planning. A multi-disciplinary team performed the analyses in order to cover as many aspects of the primary studies as possible. We found that merely implementing an HIS will not automatically increase organizational efficiency. Strategic, tactical, and operational actions have to be taken into consideration, including management involvement, integration in healthcare workflow, establishing compatibility between software and hardware and, most importantly, user involvement, education and training. The results should be interpreted as a high-order scheme, and not a predictive theory.

A quantitative analysis of the impact of a computerised information system on nurses' clinical practice using a realistic evaluation framework

OBJECTIVE

To explore nurses' perceptions of the impact on clinical practice of the use of a computerised hospital information system.

DESIGN

A realistic evaluation design based on Pawson and Tilley's work has been used across all the phases of the study. This is a theory-driven approach and focuses evaluation on the study of what works, for whom and in what circumstances. These relationships are constructed as context-mechanisms-outcomes (CMO) configurations.

MEASUREMENTS

A questionnaire was distributed to all nurses working in in-patient units of a university hospital in Spain (n=227). Quantitative data were analysed using SPSS 13.0. Descriptive statistics were used for an overall overview of nurses' perception. Inferential analysis, including both bivariate and multivariate methods (path analysis), was used for cross-tabulation of variables searching for CMO relationships.

RESULTS

Nurses (n=179) participated in the study (78.8% response rate). Overall satisfaction with the IT system was positive. Comparisons with context variables show how nursing units' context had greater influence on perceptions than users' characteristics. Path analysis illustrated that the influence of unit context variables are on outcomes and not on mechanisms.

CONCLUSION

Results from the study looking at subtle variations in users and units provide insight into how important professional culture and working practices could be in IT (information technology) implementation. The socio-technical approach on IT systems evaluation suggested in the recent literature appears to be an adequate theoretical underpinning for IT evaluation research. Realistic evaluation has proven to be an adequate method for IT evaluation.

The multimethod evaluation of a nursing information system in taiwan

The purpose of this study was to evaluate the impact of a nursing information system on nurses' daily practice. A multimethod evaluation was used that combined both quantitative and qualitative approaches to explore the influence of this technology from different perspectives. The data collection methods included the following: an evaluation scale of the system function, focus group interviews about the users' experiences, and work sampling observation of their work pattern. The data were collected from October to December 2006. The data analysis contained 623 valid questionnaires, 24 nurses' interview content, and 22830 observed nursing activities. Overall, the results indicated that nurses gave a positive evaluation of the technology in their work routine. However, they were concerned about the inadequacy of the hardware devices and response time, training and workflow change, and personal interactions with physicians and patients. Nurses on the night shift spent more time on documentation than on any other nursing activities during the three shifts. The users' technology adoption could be attributed to device sufficiency and function, content design, and workflow efficiency. It is recommended that organizations provide sufficient computer devices, ensure the content design for nursing documentation, and implement expectations for interdisciplinary cooperation.

Virtual organization of hospital medical imaging: a user satisfaction survey

A virtual medical imaging department is an innovative and demanding organizational model, to the extent that the underlying goal is to achieve a continuous and advanced organizational integration of human and physical resources, clinical data, and clienteles. To better understand the kind of benefits offered, we conducted a survey of three groups of users--radiologists, radiological technologists, and medical specialists--working in a five-site virtual organization. We received 127 valid questionnaires, for an overall response rate of 66%. The assessments vary according to the use made of the system. The scores for system quality and the quality of the data produced were markedly higher for intra-hospital use (respectively 7.9 and 8.7 out of 10) than for inter-hospital use (5.4 and 7.0). Despite the negative assessments they made of inter-hospital use, users maintained a positive attitude toward some type of virtual organization of medical imaging. Indeed, the score for Overall satisfaction with the system was very high, 8.9 out of 10. Moreover, the scores for Intended future use of the system were very high for both intra-hospital use (8.9) and inter-hospital use (8.7). We also found significant differences in perceptions among user groups.

Evaluating User Satisfaction with an Electronic Prescription System in a Primary Care Group

Introduction: Electronic prescribing has been proposed as an important strategy to reduce medication errors, improve the quality of patient care and create savings in health care costs. Despite these potential advantages, user satisfaction plays a significant role in the success of its implementation. Hence, this study aims to examine users’ satisfaction and factors associated with satisfaction regarding an electronic prescription system implemented in the National Healthcare Group Polyclinics in Singapore. Materials and Methods: An anonymous survey was administered in October 2007 to all physicians, pharmacists and pharmacy technicians working in the 9 National Healthcare Group Polyclinics. Results: Respondents included 118 doctors and 61 pharmacy staff. The overall level of satisfaction with electronic prescribing was high. Doctors and pharmacists reported a high degree of agreement that electronic prescribing reduces prescribing errors and interventions, and they did not want to go back to the paper-based system. Users were generally satisfied with the functionality of the system but there was some degree of workflow interference particularly for the pharmacy staff. Only 56.9% of the pharmacy respondents expressed satisfaction with the review function of the electronic prescription system and only 51.8% and 60% were satisfied when processing prescriptions that included items to be purchased from an external pharmacy or prescriptions with amendments. The results also revealed that satisfaction with the system was more associated with users’ perceptions about the electronic prescription system’s impact on productivity than quality of care. Conclusion: The survey results indicate that the implementation of the electronic prescription system has gone reasonably well. The survey findings provide opportunities for system and workflow enhancement, which is important as these issues could affect the acceptability of a new technology and the speed of diffusion within an organisation. Key words: Information technology, Primary healthcare, Prescriptions

Professional Practice and Innovation: Users' Attitudes to an Electronic Medical Record System and its Correlates: A Multivariate Analysis

Implementation of an electronic medical record (EMR) system increases efficiency of health services, quality of care and patient satisfaction. Successful implementation depends on many factors, one of which is how users respond to the new system. We studied medical receptionists' appraisal of the newly implemented EMR system in primary healthcare centres in Kuwait. Four hundred receptionists were selected randomly from different healthcare centres and asked to complete a user interaction satisfaction questionnaire relating to their experience of the new system. The response rate was 80.5%. A large majority of the respondents considered the system to be flexible (83%), easy (89%), and satisfying (81%). However, more than one third of the respondents (36%) found the system inadequate. Bivariate and multivariate analyses found age, typing ability, ease of data entry and computer error as significant correlates with overall user response. These findings relating to users' reactions to various aspects of the EMR should assist policymakers to recognise the causes of dissatisfaction with the EMR among medical receptionists at health centre clinics that may adversely affect its successful implementation and regular use, as well as the quality of care provided by the clinics. In addition, the findings provide information to assist the development of guidelines for future implementation of the EMR system at the secondary healthcare level.

A health examination system integrated with clinical decision support system

Health examinations play a key role in preventive medicine. We propose a health examination system named Health Examination Automatic Logic System (HEALS) to assist clinical workers in improving the total quality of health examinations. Quality of automated inference is confirmed by the zero inference error where during 6 months and 14,773 cases. Automated inference time is less than one second per case in contrast to 2 to 5 min for physicians. The most significant result of efficiency evaluation is that 3,494 of 4,356 (80.2%) cases take less than 3 min per case for producing a report summary. In the evaluation of effectiveness, novice physicians got 18% improvement in making decisions with the assistance of our system. We conclude that a health examination system with a clinical decision system can greatly reduce the mundane burden on clinical workers and markedly improve the quality and efficiency of health examination tasks.

Health IT success and failure: recommendations from literature and an AMIA workshop

With the United States joining other countries in national efforts to reap the many benefits that use of health information technology can bring for health care quality and savings, sobering reports recall the complexity and difficulties of implementing even smaller-scale systems. Despite best practice research that identified success factors for health information technology projects, a majority, in some sense, still fail. Similar problems plague a variety of different kinds of applications, and have done so for many years. Ten AMIA working groups sponsored a workshop at the AMIA Fall 2006 Symposium. It was entitled "Avoiding The F-Word: IT Project Morbidity, Mortality, and Immortality" and focused on this under-addressed problem. PARTICIPANTS discussed communication, workflow, and quality; the complexity of information technology undertakings; the need to integrate all aspects of projects, work environments, and regulatory and policy requirements; and the difficulty of getting all the parts and participants in harmony. While recognizing that there still are technical issues related to functionality and interoperability, discussion affirmed the emerging consensus that problems are due to sociological, cultural, and financial issues, and hence are more managerial than technical. Participants drew on lessons from experience and research in identifying important issues, action items, and recommendations to address the following: what "success" and "failure" mean, what contributes to making successful or unsuccessful systems, how to use failure as an enhanced learning opportunity for continued improvement, how system successes or failures should be studied, and what AMIA should do to enhance opportunities for successes. The workshop laid out a research agenda and recommended action items, reflecting the conviction that AMIA members and AMIA as an organization can take a leadership role to make projects more practical and likely to succeed in health care settings.

Redesigning care processes using an electronic health record: a system's experience

BACKGROUND

Implementation of health information technology (HIT) has encountered many difficulties and produced mixed outcomes. Yet Trinity Health, a major integrated delivery system, successfully leveraged implementation of a systemwide electronic health record (EHR) to promote process redesign and continuous quality improvement.

IMPLEMENTING A SYSTEMWIDE EHR

After several years of planning, two waves of EHR implementation were launched, in 2001 and 2003. One system HIT team collaborated with each hospital team for 18 months before its 24-hour transition to the EHR. During EHR planning, the system HIT team used five principles of redesign of care processes: (1) identify and address safety problems, (2) promote evidence-based practices, (3) reduce practice variations and standardize terminologies and care processes, (4) improve communication and relationships among clinician roles, and (5) augment multiple uses of data in HIT-supported care processes. Patient-centered work flows were developed to design improved patient care processes for different types of patients, such as medical inpatients and emergency outpatients. These admission-to-discharge work flows addressed gaps in quality, safety, and efficiency and helped ensure that the EHR and decision supports reflected crucial interactions among clinicians and with the patient. By the end of 2008, 13 of Trinity Health's 17 major health care organizations ("ministries") made the transformation to using EHRs.

DISCUSSION

EHR-supported care redesign requires development of substantial system capacities in clinical informatics, customization and standardization of vendor's products, collaboration and coordination between system and hospital implementation teams, quality training for clinicians and change agents, and significant clinician participation in local preparations.

Trends in primary care clinician perceptions of a new electronic health record

BACKGROUND

Clinician perceptions of a newly implemented electronic health record play an important role in its success or failure.

OBJECTIVE

To measure changes in primary care clinician attitudes toward an electronic health record during the first year following implementation.

DESIGN

Longitudinal survey.

PARTICIPANTS

86 primary care clinicians surveyed between December 2006 and January 2008.

MEASUREMENTS

Perceived impact on overall quality of care, patient safety, communication, and efficiency at 1, 3, 6, and 12 months following implementation.

RESULTS

Response rates for months 1, 3, 6, and 12 were 92%, 95%, 90%, and 82%, respectively. The proportion of clinicians agreeing that the EHR improved the overall quality of care (63% to 86%; p < 0.001), reduced medication-related errors (72% to 81%; p = 0.03), improved follow-up of test results (62% to 87%; p < 0.001), and improved communication among clinicians (72% to 93%; p < 0.001) increased from month 1 to month 12. During the same time period, a decreasing proportion of clinicians agreed that the EHR reduced the quality of patient interactions (49% to 33%; p = 0.001), resulted in longer patient visits (68% to 51%; p = 0.001), and increased time spent on medical documentation (78% to 68%; p = 0.006). Significant improvements in perceptions related to test result follow-up were first detected at 6 months, while those related to overall quality, efficiency, and communication were first identified at 12 months.

CONCLUSIONS

Primary care clinicians report increasingly positive perceptions of a new electronic health record within 1 year of implementation across a spectrum of domains of care.

A technology ecosystem perspective on hospital management information systems: lessons from the health literature

Hospital managers have a large range of information needs including quality metrics, financial reports, access information needs, educational, resourcing and decision support needs. Currently these needs involve interactions by managers with numerous disparate systems, both electronic such as SAP, Oracle Financials, PAS' (patient administration systems) like HOMER, and relevant websites; and paper-based systems. Hospital management information systems (HMIS) can be thought of sitting within a Technology Ecosystem (TE). In addition, Hospital Management Information Systems (HMIS) could benefit from a broader and deeper TE model, and the HMIS environment may in fact represents its own TE (the HMTE). This research will examine lessons from the health literature in relation to some of these issues, and propose an extension to the base model of a TE.

Usability testing of a web-based tool for managing open shifts on nursing units

Using observational and interview approaches, we evaluated the usability of a web-based communication tool that allows managers to announce open work shifts and staff to request work shifts. We used Morae software to record screen shots and study participants' (4 Nurse Managers, 3 Registered Nurses and 3 Patient Care Technicians) utterances during system use. Although features and functions used and patterns of use varied by role and individual, all participants found the system easy to use. Use of a particular function depended upon how effective the user perceived the function to be for the task that they wished to accomplish. User satisfaction was achieved when an effective function also allowed them to meet their task goal efficiently.

A framework for assessing climate change impacts on water and watershed systems

In this article we present a framework for assessing climate change impacts on water and watershed systems to support management decision-making. The framework addresses three issues complicating assessments of climate change impacts--linkages across spatial scales, linkages across temporal scales, and linkages across scientific and management disciplines. A major theme underlying the framework is that, due to current limitations in modeling capabilities, assessing and responding to climate change should be approached from the perspective of risk assessment and management rather than as a prediction problem. The framework is based generally on ecological risk assessment and similar approaches. A second theme underlying the framework is the need for close collaboration among climate scientists, scientists interested in assessing impacts, and resource managers and decision makers. A case study illustrating an application of the framework is also presented that provides a specific, practical example of how the framework was used to assess the impacts of climate change on water quality in a mid-Atlantic, U.S., watershed.

Hospital characteristics associated with highly automated and usable clinical information systems in Texas, United States

Background

A hospital's clinical information system may require a specific environment in which to flourish. This environment is not yet well defined. We examined whether specific hospital characteristics are associated with highly automated and usable clinical information systems.

Methods

This was a cross-sectional survey of 125 urban hospitals in Texas, United States using the Clinical Information Technology Assessment Tool (CITAT), which measures a hospital's level of automation based on physician interactions with the information system. Physician responses were used to calculate a series of CITAT scores: automation and usability scores, four automation sub-domain scores, and an overall clinical information technology (CIT) score. A multivariable regression analysis was used to examine the relation between hospital characteristics and CITAT scores.

Results

We received a sufficient number of physician responses at 69 hospitals (55% response rate). Teaching hospitals, hospitals with higher IT operating expenses (>$1 million annually), IT capital expenses (>$75,000 annually) and hospitals with larger IT staff (≥ 10 full-time staff) had higher automation scores than hospitals that did not meet these criteria (p < 0.05 in all cases). These findings held after adjustment for bed size, total margin, and ownership (p < 0.05 in all cases). There were few significant associations between the hospital characteristics tested in this study and usability scores.

Conclusion

Academic affiliation and larger IT operating, capital, and staff budgets are associated with more highly automated clinical information systems.

Expanding multi-disciplinary approaches to healthcare information technologies: what does information systems offer medical informatics?

The effective use of information technology (IT) is a crucial component for the delivery of effective services in health care. Current approaches to medical informatics (MI) research have significantly contributed to the success of IT use in health care but important challenges remain to be addressed. We believe that expanding the multi-disciplinary basis for MI research is important to meeting these research challenges. In this paper, we outline theories and methods used in information systems (IS) research that we believe can inform our understanding of health care IT applications and outcomes. To do so, we discuss some general differences in the focus and methods of MI and IS research to identify broad opportunities. We then review conceptual and methodological approaches in IS that have been applied in health care IT research. These include: technology-use mediation, collaborative work, genre theory, interpretive research, action research, and modeling. Examples of these theories and methods in healthcare IS research are illustrated.

Investigating evaluation frameworks for health information systems

BACKGROUND AND PURPOSE

Evaluation of health information systems (HIS) enables the assessment of the extent to which HIS are fulfilling their objectives in supporting the services of healthcare delivery. This paper presents an overview of evaluation in health informatics and information systems.

METHODS

Literature review on discourses, dimensions and methods of HIS and IS evaluation. A critical appraisal of selected HIS and IS evaluation frameworks is undertaken in order to identify HIS evaluation dimensions and measures. The frameworks are compared based on their inclusion of human, organizational and technological factors.

RESULTS

We found that an increasing number of evaluation studies deal with two distinct trends of HIS: one considers human and organizational issues and the other is concerned with the employment of a subjectivist approach. Our review indicates that current evaluation methods complement each other in that they evaluate different aspects of HIS and they can be improved upon.

CONCLUSIONS

Evaluation is complex; it is easy to measure many things but not necessarily the right ones. Nevertheless, it is possible to consider, a HIS evaluation framework with more comprehensive and specific measures that would incorporate technological, human and organizational issues to facilitate HIS evaluation.

Two-stage evaluation of the impact of a nursing information system in Taiwan

PURPOSE

This study evaluated the post-implementation impact of a nursing information system and identified issues related to the technology adoption process. Given the high level of investment necessary to implement information systems, evaluation has become vital to ensure successful adoption and use. Improved understanding of implementation difficulties/barriers and factors leading to them can serve as a platform for the development of strategies and education programs for users.

METHOD

The study design was a two-stage data comparison analysis of a nursing information system focusing on computerized nursing care plan use. Data were collected from nurses by questionnaire during the first month (December 2004) and 1 year after system installation (December 2005).

RESULTS

While nurses gave the information technology a higher evaluation score at the second stage than at the early stage, the overall ratings were slightly negative at both stages. Age and pressure to use the system were critical factors at both stages, whereas computer skills and perceived time using the system were vital at the beginning stage of implementation. Issues of concern at both stages were system functionality, efficiency, usability, and user support.

CONCLUSION

Implementation of an information system requires consideration of issues involving hardware and software, staff training, organizational policy, and workflow changes.

Nurses' views on the use, quality and user satisfaction with electronic medical records: questionnaire development

AIM

This paper is a report of the development of an instrument to measure nurses' views on the use, quality and user satisfaction with electronic medical records systems.

BACKGROUND

Use of electronic medical records systems in hospitals is steadily increasing, yet no validated instruments have assessed the effectiveness of these systems from the viewpoint of nurses.

METHOD

Items were designed following a literature review based on three main constructs: use, quality and user satisfaction with electronic medical records. Reliability and validity were examined based on responses from 1,666 nurses from 42 hospitals in Japan in February 2006. Exploratory factor analysis was conducted to determine the degree to which each item within a construct was associated. The reliability of each resultant factor was computed using Cronbach's alpha coefficient. Content validity was addressed by basing the items on previous surveys and review of the instrument by a panel of nurses experienced in nursing informatics. Construct validity was examined through factor analysis and correlational analyses.

FINDINGS

Extent of 'use' of electronic medical records resulted into three factors with good factor loadings, but only two had acceptable reliability. 'Quality' of electronic medical records had two factors with good factor loadings and reliability. 'User satisfaction' with electronic medical records had three factors, but only one had acceptable reliability. 'Use' and 'quality' constructs were positively correlated with 'user satisfaction'.

CONCLUSION

The final instrument incorporates 34 items from the original 44-item pool. Initial validity results were positive and therefore the instrument can be used in evaluating electronic medical records in hospitals.

Outcomes assessment of clinical information system implementation: a practical guide

Healthcare information systems (HIS) play a vital role in quality of care and the organization's daily operations. Consequently, increasing numbers of clinicians have been involved in HIS implementation, particularly for clinical information systems (CIS). Implementation of these systems is a major organizational investment, and its outcomes must be assessed. The purpose of this article is to provide clinicians and frontline informaticians with a practical guide to assess these outcomes, focusing on outcome variables, assessment methods, and timing of assessment. Based on in-depth literature reviews and their empirical experiences, the authors identified 3 frequently used outcomes: user satisfaction, clinical outcomes, and financial impact. These outcomes have been assessed employing various methods, including randomized controlled trials, pre- and post-test studies, time and motion studies, surveys, and user testing. The timing for outcomes assessments varied depending on several factors, such as learning curves or patients conditions. In conclusion, outcomes assessment is essential for the success of healthcare information technology, and the CIS implementation team members must be prepared to conduct and/or facilitate these studies.

Review: evaluating information systems in nursing

AIMS

To review existing nursing research on inpatient hospitals' information technology (IT) systems in order to explore new approaches for evaluation research on nursing informatics to guide further design and implementation of effective IT systems.

BACKGROUND

There has been an increase in the use of IT and information systems in nursing in recent years. However, there has been little evaluation of these systems and little guidance on how they might be evaluated.

METHODS

A literature review was conducted between 1995 and 2005 inclusive using CINAHL and Medline and the search terms 'nursing information systems', 'clinical information systems', 'hospital information systems', 'documentation', 'nursing records', 'charting'.

RESULTS

Research in nursing information systems was analysed and some deficiencies and contradictory results were identified which impede a comprehensive understanding of effective implementation. There is a need for IT systems to be understood from a wider perspective that includes aspects related to the context where they are implemented.

CONCLUSIONS

Social and organizational aspects need to be considered in evaluation studies and realistic evaluation can provide a framework for the evaluation of information systems in nursing.

RELEVANCE TO CLINICAL PRACTICE

The rapid introduction of IT systems for clinical practice urges evaluation of already implemented systems examining how and in what circumstances they work to guide effective further development and implementation of IT systems to enhance clinical practice. Evaluation involves more factors than just involving technologies such as changing attitudes, cultures and healthcare practices. Realistic evaluation could provide configurations of context-mechanism-outcomes that explain the underlying relationships to understand why and how a programme or intervention works.

An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit)

BACKGROUND AND PURPOSE

The realization of Health Information Systems (HIS) requires rigorous evaluation that addresses technology, human and organization issues. Our review indicates that current evaluation methods evaluate different aspects of HIS and they can be improved upon. A new evaluation framework, human, organization and technology-fit (HOT-fit) was developed after having conducted a critical appraisal of the findings of existing HIS evaluation studies. HOT-fit builds on previous models of IS evaluation--in particular, the IS Success Model and the IT-Organization Fit Model. This paper introduces the new framework for HIS evaluation that incorporates comprehensive dimensions and measures of HIS and provides a technological, human and organizational fit.

METHODS

Literature review on HIS and IS evaluation studies and pilot testing of developed framework. The framework was used to evaluate a Fundus Imaging System (FIS) of a primary care organization in the UK. The case study was conducted through observation, interview and document analysis.

RESULTS

The main findings show that having the right user attitude and skills base together with good leadership, IT-friendly environment and good communication can have positive influence on the system adoption.

CONCLUSIONS

Comprehensive, specific evaluation factors, dimensions and measures in the new framework (HOT-fit) are applicable in HIS evaluation. The use of such a framework is argued to be useful not only for comprehensive evaluation of the particular FIS system under investigation, but potentially also for any Health Information System in general.

Nurses' Experiences Using a Nursing Information System

Adoption of information technology in nursing practice has become a trend in healthcare. The impact of this technology on users has been widely studied, but little attention has been given to its influence at the beginning stage of implementation. Knowing the barriers to adopting technology could shorten this transition stage and minimize its negative influences. The purpose of this study was to explore nurses' experiences in the early stage of implementing a nursing information system. Focus groups were used to collect data at a medical center in Taiwan. The results showed that nurses had problems with the system's content design, had insufficient training, were concerned about data security, were stressed by added work, and experienced poor interdisciplinary cooperation. To smooth this beginning stage, the author recommends involving nurses early in the system design, providing sufficient training in keyboard entry skills, redesigning workflow, and improving interdisciplinary communication.

A methodological and operative framework for the evaluation of an e-health project

Assessing public sector ICT investments represents the premise for successful implementation of an e-health strategy. The recent literature stresses the importance of going beyond the mere financial and/or technical dimensions of the analysis. Consequently, the paper proposes an example of e-health project evaluation aiming to develop measures which get close to the notion of benefits to the different stakeholders involved: top management, patients, local community. The case study refers to an Italian health care organization that implemented a project of digitalization of its clinical reports production few years ago. Based on on-field research, different approaches are used to assess costs and benefits from different stakeholders' perspectives. The results of a multidimensional evaluation are reported to emphasize the need for different measures to assess the sustainability of an e-health project according to the financial convenience, the social role of the organization, and the contingent situation.

Development and testing of an evaluation scale of personal digital assistants

With the increasingly popular use of PDAs in healthcare, the need for a reliable scale to evaluate this technology has become critical. The authors describe the development and testing of an evaluation scale for PDAs in nurses' daily practice. A review of the literature generated a 30-item questionnaire that was administered online to a convenience sample of 334 hospital nurses in Taiwan. The scale's reliability and validity were established by item and factor analyses. From the original pool of 30 items, 24 were selected and grouped into six major constructs: user preference, nursing efficiency, user training, device features, drug content, and user support. The scale's overall reliability (alpha coefficient) was 0.88. Survey results showed an overall slightly negative score for the evaluation scale for PDAs, indicating a slightly moderate evaluation of PDA use. Additional psychometric analysis and refinement of item wordings are recommended.

The role of perceptions of clinicians in their adoption of a web-based antibiotic approval system: do perceptions translate into actions?

PURPOSE

Computerized Decision Support Systems have been shown to improve clinicians' performance. Clinicians' adoption of these systems is crucial for their success. Studying clinicians' perceptions can provide an insight into the determinants of clinicians' adoption of such systems. The aim of this study was to measure clinicians' perceptions of ease of use and usefulness of a web-based antibiotic approval system, and to investigate the relationship between the reported perceptions and use of the system.

METHODS

Potentially identifiable coded surveys were sent to a total of 70 senior and 150 junior medical staff, and 30 pharmacists all working at a tertiary care referral teaching hospital of Melbourne, Australia. Clinicians' perceptions of ease of use and usefulness of the antibiotic approval system; clinicians' general computer use; and clinicians' usage of the antibiotic approval system were measured.

RESULTS

The overall response rate from the clinicians was 53.4%. The majority of the participants (70% and above) found it easy to obtain antibiotic approval using the system. More than 80% of the participants believed that the system will decrease the inappropriate use of antibiotics at the hospital. Clinicians who were more likely to use the system also found it easy to learn (Rho=0.392, p=0.001), easy to show others how to use the system (Rho=0.298, p=0.014), easy to find additional information (Rho=0.317, p=0.009), and easy to use it within their daily workflow (Rho=0.268, p=0.028). In addition, the clinicians were also more likely to use the system if they believed that it will improve their adherence to evidence-based practice (Rho=0.352, p=0.003).

CONCLUSION

The majority of clinicians in an independent investigation of the antibiotic approval system found the system easy to use and useful to them. A number of clinicians' perceptions about the system were found to be correlated with the actual usage of the system by the clinicians.

Acceptance and cognitive load in a clinical setting of a novel device allowing natural real-time data acquisition

INTRODUCTION

This paper reports the findings of an evaluation study in the field of human-computer interaction about the use of a new data acquisition device, the digital pen. It focuses on specific aspects of the interaction between the users and the technology: the cognitive burden induced by the design of the tool in real conditions of use and its impact on user acceptance.

METHODOLOGY

Human cognition is embedded in a complex sociocultural world. Therefore, we opted for ethnographically informed investigations reinforced by a satisfaction survey. The work context chosen for these investigations was the emergency room triage process.

RESULTS

The technology meets a high acceptance (median 3 on a [-5,5] scale) shaded by unexpected additional cognitive burdens. These burdens originate in several technological and ergonomic flaws that have been discovered during the observations. These results have been used to improve the technology.

CONCLUSION

We demonstrate the importance of this kind of field study to uncover unexpected possible sources of failure of acceptance of a new technology. Such kind of study should be held prior to the introduction of a new technology to lower the common failure rate encountered in the field of medical informatics.

Nursing administrators' experiences in managing PDA use for inpatient units

The adoption of information technology in patient care has become a trend in healthcare organizations. The impact of this technology on end users has been widely studied, but little attention has been given to its influence from a management perspective. The purpose of this study was to explore nurse managers' perceived experiences in implementing a policy to adopt personal digital assistant technology. A descriptive, exploratory qualitative approach (one-on-one, in-depth interviews) was used to collect data from 16 nurse managers of inpatient units at a medical center in Taiwan. Interview data were analyzed according to Miles and Huberman's data reduction, data display, and conclusion verification process. The results revealed that nurse managers experienced the limitations of technology, training issues, doctors' obstructive influence, role conflict, and improvement of future personal digital assistant use. These results can be used to improve strategic organizational planning and in-service training programs to implement information systems.

From the front line, report from a near paperless hospital: mixed reception among health care professionals

OBJECTIVE

Many Norwegian hospitals that are equipped with an electronic medical record (EMR) system now have proceeded to withdraw the paper-based medical record from clinical workflow. In two previous survey-based studies on the effect of removing the paper-based medical record on the work of physicians, nurses and medical secretaries, we concluded that to scan and eliminate the paper based record was feasible, but that the medical secretaries were the group that reported to benefit the most from the change. To further explore the effects of removing the paper based record, especially in regard to medical personnel, we now have conducted a follow up study of a hospital that has scanned and eliminated its paper-based record.

DESIGN

A survey of 27 physicians, 60 nurses and 30 medical secretaries was conducted. The results were compared with those from a previous study conducted three years earlier at the same department.

MEASUREMENTS

The questionnaire (see online Appendix) covered the frequency of use of the EMR system for specific tasks by physicians, nurses and medical secretaries, the ease of performing these tasks compared to previous routines, user satisfaction and computer literacy.

RESULTS

Both physicians and nurses displayed increased use of the EMR compared to the previous study, while medical secretaries reported generally unchanged but high use.

CONCLUSION

The increase in use was not accompanied by a similar change in factors such as computer literacy or technical changes, suggesting that these typical success factors are necessary but not sufficient.

The N-CODES Project Moves to User Testing

For a technology to be implemented successfully, it must be thoroughly tested by the stakeholders who will use it in practice. This article reports on the user testing of the Nurse Computer Decision Support working prototype. Ten acute care nurses with varying levels of education and experience participated in the testing. The study's protocol consisted of a series of tasks requiring the participants to use the system while solving patient problems. Both quantitative and qualitative data on usability, ease of navigation, and nurse satisfaction were collected. The nurses ranked usability and ease of navigation highly. Nurses were excited about the system's clinical potential and felt that it would be useful to all practitioners regardless of experience. Testing the system in an actual clinical setting is the next step, but it is clear that the system has extraordinary potential to improve clinical decision making and patient outcomes.

A risk assessment of two interorganizational clinical information systems

A risk analysis framework was used to examine the implementation barriers that may hamper the successful implementation of interorganizational clinical information systems (ICIS). In terms of study design, an extensive literature review was first performed in order to elaborate a comprehensive model of project risk factors. To test the applicability of the model, we next conducted a longitudinal multiple-case study of two large-scale ICIS demonstration projects carried out in Quebec, Canada. Variations in the levels of several risk dimensions measured throughout the duration of the projects were analyzed to determine their impact on successful implementation. The analysis shows that the proposed framework, composed of five risk dimensions, was very robust, and suitable for conducting a thorough risk analysis. The results also show that there are links between the quality of the risk management and the level of project outcomes. To be successful, it is important that the implementation efforts be distributed proportionally according to the importance of each of the risk factors. Furthermore, because the risks evolve dynamically, there is a need for high responsiveness to emerging implementation problems. Thus, implementation success lies in the ability of the project management team to be aware of and to manage several risk threats simultaneously and coherently since they evolve dynamically through time and interact with one another.

Clinical information technology capabilities in four U.S. hospitals: testing a new structural performance measure

BACKGROUND

Few tools exist to quantify the performance of a hospital's information system from a user perspective.

OBJECTIVES

Our objective was to develop and evaluate a survey-based metric that assesses the automation and usability of a hospital's information system.

RESEARCH DESIGN AND METHODS

This is a cross-sectional study of 117 physicians and 3 chief information officers (CIOs) working in 2 community hospitals with historically low investment in IT (Hospitals A and B), an academic hospital with an advanced IT system (Hospital C), or a major Veterans Affairs hospital (Hospital D). Respondents completed a survey assessing their institution's information system. The mean of 90 summed responses yields the clinical information technology (CIT) index, a global measure of a hospital's information system performance on a 100-point scale.

RESULTS

On the global CIT index, mean physician scores were significantly higher for hospitals with advanced IT (61.1 and 64.3 for C and D) compared with those with low investment in IT (32.6 and 29.4 for A and B, P < 0.001). These differences also were observed for each of 7 separate subdomains. The CIO scores, 74.7, 78.0 for Hospitals C and D, and 44.5 for Hospitals A and B, paralleled the mean physician scores for these hospitals. All measures exhibited low variance for each hospital (eg, standard deviations for the CIT index ranged from 5.9 to 8.1) and intraclass correlation was high (Chronbach's alpha >.70).

CONCLUSIONS

This assessment tool demonstrates initial evidence of validity and reliability.

An electronic medical record in primary care: impact on satisfaction, work efficiency and clinic processes

User satisfaction with an electronic medical record (EMR) plays a decisive role in its implementation and subsequent use. We developed a survey tool to identify features of an EMR that contribute to user satisfaction and administered it in an adult primary care clinic. Most physician respondents were highly satisfied with the EMR and used all of its components. The EMR decreased the time to develop a synopsis of the patient and improved communication efficiency. Most respondents valued remote access to the EMR. Electronic messaging was an important component of improved care delivery according to 80% of the respondents. Access to online references within the EMR was not valued over web-based access for most respondents. Our results demonstrate acceptance of an EMR in adult primary care. Features such as remote access and electronic messaging were surprisingly useful and successful for primary care practice.

Determinants of user satisfaction with a Clinical Information System

Clinical Information Systems (CIS) implementation has faced user resistance. Consequently, we aimed to assess the acceptability of an integrated CIS. We designed an electronic survey instrument from two theoretical models (Delone and McLean, and Technology Acceptance Model). Dimensions hypothesized to be determinant of user satisfaction were: user characteristics, CIS use, quality, usefulness, and service quality. The questionnaire was administered to physicians, nurses and medical secretaries of the Georges Pompidou university Hospital (HEGP) in Paris. Answers were obtained from 324 users (93 physicians, 174 nurses, and 57 secretaries). Cronbach's alpha coefficients showed a correct reliability within each dimension. Secretaries and nurses were more satisfied with the CIS than physicians. Except for CIS use, after adjustment for confounders, female gender, perceived CIS quality, usefulness, and service quality were strongly correlated with user satisfaction. This study reinforces the necessity of several models and dimensions to evaluate the acceptability of a complex CIS, with a specific approach for different user profiles.

Designing an EMR planning process based on staff attitudes toward and opinions about computers in healthcare

Electronic medical record (EMR) systems have been in use for more than 2 decades. Studies documenting nursing satisfaction with an EMR system, the benefits of an EMR, implementation barriers, user acceptance, the importance of staff buy-in, and the importance of attitudes toward and expectations from user buy-in are in the literature. Central to many studies is the importance of nursing staff buy-in to the successful implementation and ongoing use of an EMR, as well as the dependency of buy-in on staff attitudes and expectations. Buy-in is a precursor to effective use. Consequently, staff buy-in is a prerequisite to collecting and making optimum use of the data contained in an EMR. Data collected from an EMR containing rich, accurate documentation of nursing interventions and patient responses support evidence-based practice changes and documentation of the import of the care provided by nurses.

Evaluating user interactions with clinical information systems: a model based on human-computer interaction models

OBJECTIVES

This article proposes a model for dimensions involved in user evaluation of clinical information systems (CIS). The model links the dimensions in traditional CIS evaluation and the dimensions from the human-computer interaction (HCI) perspective.

PROPOSED METHOD

In this article, variables are defined as the properties measured in an evaluation, and dimensions are defined as the factors contributing to the values of the measured variables. The proposed model is based on a two-step methodology with: (1) a general review of information systems (IS) evaluations to highlight studied variables, existing models and frameworks, and (2) a review of HCI literature to provide the theoretical basis to key dimensions of user evaluation.

RESULTS

The review of literature led to the identification of eight key variables, among which satisfaction, acceptance, and success were found to be the most referenced.

DISCUSSION

Among those variables, IS acceptance is a relevant candidate to reflect user evaluation of CIS. While their goals are similar, the fields of traditional CIS evaluation, and HCI are not closely connected. Combining those two fields allows for the development of an integrated model which provides a model for summative and comprehensive user evaluation of CIS. All dimensions identified in existing studies can be linked to this model and such an integrated model could provide a new perspective to compare investigations of different CIS systems.