Recommendations for responsible monitoring and regulation of clinical software systems. American Medical Informatics Association, Computer-based Patient Record Institute, Medical Library Association, Association of Academic Health Science Libraries, American Health Information Management Association, American Nurses Association

Toward a responsible future: recommendations for AI-enabled clinical decision support

BACKGROUND

Integrating artificial intelligence (AI) in healthcare settings has the potential to benefit clinical decision-making. Addressing challenges such as ensuring trustworthiness, mitigating bias, and maintaining safety is paramount. The lack of established methodologies for pre- and post-deployment evaluation of AI tools regarding crucial attributes such as transparency, performance monitoring, and adverse event reporting makes this situation challenging.

OBJECTIVES

This paper aims to make practical suggestions for creating methods, rules, and guidelines to ensure that the development, testing, supervision, and use of AI in clinical decision support (CDS) systems are done well and safely for patients.

MATERIALS AND METHODS

In May 2023, the Division of Clinical Informatics at Beth Israel Deaconess Medical Center and the American Medical Informatics Association co-sponsored a working group on AI in healthcare. In August 2023, there were 4 webinars on AI topics and a 2-day workshop in September 2023 for consensus-building. The event included over 200 industry stakeholders, including clinicians, software developers, academics, ethicists, attorneys, government policy experts, scientists, and patients. The goal was to identify challenges associated with the trusted use of AI-enabled CDS in medical practice. Key issues were identified, and solutions were proposed through qualitative analysis and a 4-month iterative consensus process.

RESULTS

Our work culminated in several key recommendations: (1) building safe and trustworthy systems; (2) developing validation, verification, and certification processes for AI-CDS systems; (3) providing a means of safety monitoring and reporting at the national level; and (4) ensuring that appropriate documentation and end-user training are provided.

DISCUSSION

AI-enabled Clinical Decision Support (AI-CDS) systems promise to revolutionize healthcare decision-making, necessitating a comprehensive framework for their development, implementation, and regulation that emphasizes trustworthiness, transparency, and safety. This framework encompasses various aspects including model training, explainability, validation, certification, monitoring, and continuous evaluation, while also addressing challenges such as data privacy, fairness, and the need for regulatory oversight to ensure responsible integration of AI into clinical workflow.

CONCLUSIONS

Achieving responsible AI-CDS systems requires a collective effort from many healthcare stakeholders. This involves implementing robust safety, monitoring, and transparency measures while fostering innovation. Future steps include testing and piloting proposed trust mechanisms, such as safety reporting protocols, and establishing best practice guidelines.

Evolving Software Architecture Design in Telemedicine: A PRISMA-based Systematic Review

OBJECTIVES

This article presents a systematic review of recent advancements in telemedicine architectures for continuous monitoring, providing a comprehensive overview of the evolving software engineering practices underpinning these systems. The review aims to illuminate the critical role of telemedicine in delivering healthcare services, especially during global health crises, and to emphasize the importance of effectiveness, security, interoperability, and scalability in these systems.

METHODS

A systematic review methodology was employed, adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework. As the primary research method, the PubMed, IEEE Xplore, and Scopus databases were searched to identify articles relevant to telemedicine architectures for continuous monitoring. Seventeen articles were selected for analysis, and a methodical approach was employed to investigate and synthesize the findings.

RESULTS

The review identified a notable trend towards the integration of emerging technologies into telemedicine architectures. Key areas of focus include interoperability, security, and scalability. Innovations such as cognitive radio technology, behavior-based control architectures, Health Level Seven International (HL7) Fast Healthcare Interoperability Resources (FHIR) standards, cloud computing, decentralized systems, and blockchain technology are addressing challenges in remote healthcare delivery and continuous monitoring.

CONCLUSIONS

This review highlights major advancements in telemedicine architectures, emphasizing the integration of advanced technologies to improve interoperability, security, and scalability. The findings underscore the successful application of cognitive radio technology, behavior-based control, HL7 FHIR standards, cloud computing, decentralized systems, and blockchain in advancing remote healthcare delivery.

The Quantum-Medical Nexus: Understanding the Impact of Quantum Technologies on Healthcare

In a world characterized by rapid technological evolution, the integration of quantum technologies into the realm of healthcare has emerged as a transformative force. This narrative review explores the journey of quantum innovations in medicine, delving into the fundamental principles of quantum mechanics that underpin quantum computing, sensing, and communication. From the birth of quantum theory to the advent of practical quantum applications, we journey through historical milestones that have paved the way for a quantum-powered future in healthcare. The narrative unfolds to reveal the profound implications of quantum technologies in healthcare, ranging from accelerated drug discovery and genomic analysis to secure data transmission and telemedicine. Real-world case studies illuminate successful applications, while the review addresses the ethical, societal, and regulatory considerations that accompany this quantum revolution. As we peer into the future, we contemplate the challenges that lie ahead and offer recommendations for researchers and policymakers to forge a harmonious and equitable synergy between quantum and medicine. In a world where innovation outpaces the tick of the clock, this narrative review serves as a timely guide for those poised to shape the quantum healthcare landscape, where precision and compassion converge and the possibilities are limitless.

Recommendations for the safe, effective use of adaptive CDS in the US healthcare system: an AMIA position paper

The development and implementation of clinical decision support (CDS) that trains itself and adapts its algorithms based on new data-here referred to as Adaptive CDS-present unique challenges and considerations. Although Adaptive CDS represents an expected progression from earlier work, the activities needed to appropriately manage and support the establishment and evolution of Adaptive CDS require new, coordinated initiatives and oversight that do not currently exist. In this AMIA position paper, the authors describe current and emerging challenges to the safe use of Adaptive CDS and lay out recommendations for the effective management and monitoring of Adaptive CDS.

Review of Reported Clinical Information System Adverse Events in US Food and Drug Administration Databases

BACKGROUND: The US FDA has been collecting information on medical devices involved in significant adverse advents since 1984. These reports have been used by researchers to advise clinicians on potential risks and complications of using these devices. OBJECTIVE: Research adverse events related to the use of Clinical Information Systems (CIS) as reported in FDA databases. METHODS: Three large, national, adverse event medical device databases were examined for reports pertaining to CIS. RESULTS: One hundred and twenty unique reports (from over 1.4 million reports) were found, representing 32 manufacturers. The manifestations of these adverse events included: missing or incorrect data, data displayed for the wrong patient, chaos during system downtime and system unavailable for use. Analysis of these reports illustrated events associated with system design, implementation, use, and support. CONCLUSION: The identified causes can be used by manufacturers to improve their products and by clinical facilities and providers to adjust their workflow and implementation processes appropriately. The small number of reports found indicates a need to raise awareness regarding publicly available tools for documenting problems with CIS and for additional reporting and dialog between manufacturers, organizations, and users.

Impact assessment on patients' satisfaction and healthcare professionals' commitment of software supporting Directly Observed Treatment, Short-course: A protocol proposal

Doctors, nurses, and other healthcare professionals use software that affects the patients. Directly Observed Treatment, Short-course is the name given to the tuberculosis control strategy recommended by the World Health Organization. The main goal of this work is to propose a protocol for evaluating the impact of healthcare software supporting Directly Observed Treatment, Short-course on patients, healthcare professionals, and services. The proposed protocol consists of a set of instruments and steps. The instruments are reliable and validated existing questionnaires to be applied before and after using the software tool. The literature points out the need for standards on the software assessment. This is particularly critical when software affects patients directly. The present protocol is a universal tool to assess the impact of software used to support the fight against the tragedy of tuberculosis where a rigorous evaluation of IT in healthcare is highly recommended and of great importance.

50 years of informatics research on decision support: what's next

OBJECTIVES

To reflect on the history, status, and future trends of decision support in health and biomedical informatics. To highlight the new challenges posed by the complexity and diversity of genomic and clinical domains. To examine the emerging paradigms for supporting cost-effective, personalized decision making.

METHODS

A group of international experts in health and biomedical informatics presented their views and discussed the challenges and issues on decision support at the Methods of Information in Medicine 50th anniversary symposium. The experts were invited to write short articles summarizing their thoughts and positions after the symposium.

RESULTS AND CONCLUSIONS

The challenges posed by the complexity and diversity of the domain knowledge, system infrastructure, and usage pattern are highlighted. New requirements and computational paradigms for representing, using, and acquiring biomedical knowledge and healthcare protocols are proposed. The underlying common themes identified for developing next-generation decision support include incorporating lessons from history, uniform vocabularies, integrative interfaces, contextualized decisions, personalized recommendations, and adaptive solutions.

The evolution of eProtocols that enable reproducible clinical research and care methods

Unnecessary variation in clinical care and clinical research reduces our ability to determine what healthcare interventions are effective. Reducing this unnecessary variation could lead to further healthcare quality improvement and more effective clinical research. We have developed and used electronic decision support tools (eProtocols) to reduce unnecessary variation. Our eProtocols have progressed from a locally developed mainframe computer application in one clinical site (LDS Hospital) to web-based applications available in multiple languages and used internationally. We use eProtocol-insulin as an example to illustrate this evolution. We initially developed eProtocol-insulin as a local quality improvement effort to manage stress hyperglycemia in the adult intensive care unit (ICU). We extended eProtocol-insulin use to translate our quality improvement results into usual clinical care at Intermountain Healthcare ICUs. We exported eProtocol-insulin to support research in other US and international institutions, and extended our work to the pediatric ICU. We iteratively refined eProtocol-insulin throughout these transitions, and incorporated new knowledge about managing stress hyperglycemia in the ICU. Based on our experience in the development and clinical use of eProtocols, we outline remaining challenges to eProtocol development, widespread distribution and use, and suggest a process for eProtocol development. Technical and regulatory issues, as well as standardization of protocol development, validation and maintenance, need to be addressed. Resolution of these issues should facilitate general use of eProtocols to improve patient care.

INVESTIGATION OF THE ESSENTIALS FOR INTEGRATING OFF-THE-SHELF INDUSTRIAL ROBOTICS IN PRECISE COMPUTER-ASSISTED SURGERY

Off-the-shelf industrial robotic technologies have achieved significant advancements in the past several decades in terms of mechanics and automation performances. We are expecting to take advantage of the industrial robots for assisting surgeons in surgeries and quick prototyping a robotic surgery system. In precise computer-assisted surgeries (CASs), such as pelvic-acetabular surgery, eye surgery, or neurosurgery, it is extremely important to position the tools accurately and precisely for surgical operations. Some of the industrial robotics arms are able to achieve good repeatability and dexterity while positioning the surgical tools. To enable the application of industrial robots in the surgical rooms, there are several other essential modules to be integrated to the robotic surgery systems, such as real-time navigation system, surgical planning system, and surgeon-guidance system. In this paper, we review the existing studies on the medical robots including the ones using industrial robots, and then investigate the essentials for using industrial robots in computer-integrated surgery.

Creating an oversight infrastructure for electronic health record-related patient safety hazards

Electronic health records (EHRs) have potential quality and safety benefits. However, reports of EHR-related safety hazards are now emerging. The Office of the National Coordinator for Health Information Technology recently sponsored an Institute of Medicine committee to evaluate how health information technology use affects patient safety. In this article, we propose the creation of a national EHR oversight program to provide dedicated surveillance of EHR-related safety hazards and to promote learning from identified errors, close calls, and adverse events. The program calls for data gathering, investigation/analysis, and regulatory components. The first 2 functions will depend on institution-level EHR safety committees that will investigate all known EHR-related adverse events and near-misses and report them nationally using standardized methods. These committees should also perform routine safety self-assessments to proactively identify new risks. Nationally, we propose the long-term creation of a centralized, nonpartisan board with an appropriate legal and regulatory infrastructure to ensure the safety of EHRs. We discuss the rationale of the proposed oversight program and its potential organizational components and functions. These include mechanisms for robust data collection and analyses of all safety concerns using multiple methods that extend beyond reporting, multidisciplinary investigation of selected high-risk safety events, and enhanced coordination with other national agencies to facilitate broad dissemination of hazards information. Implementation of this proposed infrastructure can facilitate identification of EHR-related adverse events and errors and potentially create a safer and more effective EHR-based health care delivery system.

The dangerous decade

Over the next 10 years, more information and communication technology (ICT) will be deployed in the health system than in its entire previous history. Systems will be larger in scope, more complex, and move from regional to national and supranational scale. Yet we are at roughly the same place the aviation industry was in the 1950s with respect to system safety. Even if ICT harm rates do not increase, increased ICT use will increase the absolute number of ICT related harms. Factors that could diminish ICT harm include adoption of common standards, technology maturity, better system development, testing, implementation and end user training. Factors that will increase harm rates include complexity and heterogeneity of systems and their interfaces, rapid implementation and poor training of users. Mitigating these harms will not be easy, as organizational inertia is likely to generate a hysteresis-like lag, where the paths to increase and decrease harm are not identical.

Health information technology: fallacies and sober realities

Current research suggests that the rate of adoption of health information technology (HIT) is low, and that HIT may not have the touted beneficial effects on quality of care or costs. The twin issues of the failure of HIT adoption and of HIT efficacy stem primarily from a series of fallacies about HIT. We discuss 12 HIT fallacies and their implications for design and implementation. These fallacies must be understood and addressed for HIT to yield better results. Foundational cognitive and human factors engineering research and development are essential to better inform HIT development, deployment, and use.

Challenges in ethics, safety, best practices, and oversight regarding HIT vendors, their customers, and patients: a report of an AMIA special task force

The current commercial health information technology (HIT) arena encompasses a number of competing firms that provide electronic health applications to hospitals, clinical practices, and other healthcare-related entities. Such applications collect, store, and analyze patient information. Some vendors incorporate contract language whereby purchasers of HIT systems, such as hospitals and clinics, must indemnify vendors for malpractice or personal injury claims, even if those events are not caused or fostered by the purchasers. Some vendors require contract clauses that force HIT system purchasers to adopt vendor-defined policies that prevent the disclosure of errors, bugs, design flaws, and other HIT-software-related hazards. To address this issue, the AMIA Board of Directors appointed a Task Force to provide an analysis and insights. Task Force findings and recommendations include: patient safety should trump all other values; corporate concerns about liability and intellectual property ownership may be valid but should not over-ride all other considerations; transparency and a commitment to patient safety should govern vendor contracts; institutions are duty-bound to provide ethics education to purchasers and users, and should commit publicly to standards of corporate conduct; and vendors, system purchasers, and users should encourage and assist in each others' efforts to adopt best practices. Finally, the HIT community should re-examine whether and how regulation of electronic health applications could foster improved care, public health, and patient safety.

Confronting and resolving an ethical dilemma associated with a practice based evaluation using observational methodology of health information technology

SUMMARY

As the adoption of health information technology (HIT) has escalated, efforts to evaluate its uptake have increased. The evaluation of HIT often requires direct observation of health care practitioners interacting with the system. When in the field, the evaluator who is not a trained health care provider may observe suboptimal use of the technology. If evaluators have plans to share the results of the evaluation at the conclusion of the study, they face a decision point about whether to disclose interim results and the implications of doing so. To provide HIT evaluators with guidance about what issues to weigh when observing the implementation of HIT, this paper presents a study of an actual case and discusses the following considerations: (1) whether the evaluation of HIT is considered to be human subject research; (2) if the evaluation is human subject research, whether the Institutional Review Board will consider it exempt from review or subjected to expedited or full review; and (3) how interim disclosure to the clinic management impacts the research study. The recommendations to evaluators include use of a protocol for interim disclosures to patients, clinicians, and/or clinical management for both quality assurance initiatives and human subjects research.

Evaluation of causes and frequency of medication errors during information technology downtime

PURPOSE

The causes and frequency of medication errors occurring during information technology downtime were evaluated.

METHODS

Individuals from a convenience sample of 78 hospitals who were directly responsible for supporting and maintaining clinical information systems (CISs) and automated dispensing systems (ADSs) were surveyed using an online tool between February 2007 and May 2007 to determine if medication errors were reported during periods of system downtime. The errors were classified using the National Coordinating Council for Medication Error Reporting and Prevention severity scoring index. The percentage of respondents reporting downtime was estimated.

RESULTS

Of the 78 eligible hospitals, 32 respondents with CIS and ADS responsibilities completed the online survey for a response rate of 41%. For computerized prescriber order entry, patch installations and system upgrades caused an average downtime of 57% over a 12-month period. Lost interface and interface malfunction were reported for centralized and decentralized ADSs, with an average downtime response of 34% and 29%, respectively. The average downtime response was 31% for software malfunctions linked to clinical decision-support systems. Although patient harm did not result from 30 (54%) medication errors, the potential for harm was present for 9 (16%) of these errors.

CONCLUSION

Medication errors occurred during CIS and ADS downtime despite the availability of backup systems and standard protocols to handle periods of system downtime. Efforts should be directed to reduce the frequency and length of down-time in order to minimize medication errors during such downtime.

The informatics opportunities at the intersection of patient safety and clinical informatics

Health care providers have a basic responsibility to protect patients from accidental harm. At the institutional level, creating safe health care organizations necessitates a systematic approach. Effective use of informatics to enhance safety requires the establishment and use of standards for concept definitions and for data exchange, development of acceptable models for knowledge representation, incentives for adoption of electronic health records, support for adverse event detection and reporting, and greater investment in research at the intersection of informatics and patient safety. Leading organizations have demonstrated that health care informatics approaches can improve safety. Nevertheless, significant obstacles today limit optimal application of health informatics to safety within most provider environments. The authors offer a series of recommendations for addressing these challenges.

Downtime procedures for a clinical information system: a critical issue

As computers become embedded in clinical workflow processes, disruptions to access can have serious consequences. The Health Evaluation through Logical Processing system at LDS Hospital is a computerized hospital information system that has been under continuous development for more than 30 years. The system maintains a 99.85% uptime and averages more than 17,000 logons per day. The first formal downtime plan for this system was developed in 1992 in anticipation of a major hardware installation. In early 2000 after a series of planned downtimes from which we did not recover smoothly, our Software Oversight Committee became interested in understanding downtime procedures. A downtime plan for clinical users was developed and tested and is discussed. A March 2000 downtime survey of 103 clinical staff provided additional information to refine the plan. The downtime plan now includes explicit instructions about the clinical data that must be reentered after a downtime and also includes a plan for a regularly scheduled downtime practice drill similar to a fire drill.

Rethinking electronic health records to better achieve quality and safety goals

Health care information technology changes the ecosystem of a practice. Human roles, process work flow, and technology infrastructure are tightly interrelated. Medical errors may increase if a change in one is not accommodated by a change in the others. Introduction of information technology should be approached as an iterative process of care improvement rather than as a one-time insertion of an information system into established practice. Information technology supports a family of technological approaches, each with distinct mechanisms of action, benefits, and side effects. By matching technological approach to task and staging introduction into practice, initial benefit can be obtained more quickly, at reduced cost, while managing risk of a misfit. A staged approach to turning direct access by patients to their health information into more effective care is presented as an example of this strategy.

A roadmap for national action on clinical decision support

This document comprises an AMIA Board of Directors approved White Paper that presents a roadmap for national action on clinical decision support. It is published in JAMIA for archival and dissemination purposes. The full text of this material has been previously published on the AMIA Web site (www.amia.org/inside/initiatives/cds). AMIA is the copyright holder.

To decay is system: the challenges of keeping a health information system alive

Health information system (HIS) architecture and socio-technical approaches for system deployment have been topics of systematic research for decades. Sustainable operation in gradually changing environments, however, has not yet received sufficient attention. Even HIS that have gone life to the satisfaction of their developers and end-users may degrade gracefully or fail catastrophically if not continuously and thoroughly kept in sync with their environment. Critical environmental changes may owe their origins to the complexity of health care and its delivery. Seemingly minor environmental changes can result in significant failures on the part of the information system and may adversely affect the quality of health care delivered. Such minor degradation or near failure may go unnoticed for a while and then hit unexpectedly. Five origins of decay will be analyzed. Methods of systematic observation and containment of such decaying processes will tentatively be presented. Some origins of system decay exist in the immediate hospital or regional setting of usage. Indicators to identify processes of decay will be suggested and methods to preemptively reduce the risk of decay will be presented. Other origins span national health care systems or beyond. Not all such risks can hence be controlled locally. Software Oversight Committees may be an instrument to monitor those risks that cannot be controlled through routine local management.

On exemplary scientific conduct regarding submission of manuscripts to biomedical informatics journals

As the Editors of leading international biomedical informatics journals, the authors report on a recent pattern of improper manuscript submissions to journals in our field. As a guide for future authors, we describe ethical and pragmatic issues related to submitting work for peer-reviewed journal publication. We propose a coordinated approach to the problem that our respective journals will follow. This Editorial is being jointly published in the following journals represented by the authors: Computer Methods and Programs in Biomedicine, International Journal of Medical Informatics, Journal of Biomedical Informatics, Journal of the American Medical Informatics Association, and Methods of Information in Medicine.

Healthcare software assurance

Software assurance is a rigorous, lifecycle phase-independent set of activities which ensure completeness, safety, and reliability of software processes and products. This is accomplished by guaranteeing conformance to all requirements, standards, procedures, and regulations. These assurance processes are even more important when coupled with healthcare software systems, embedded software in medical instrumentation, and other healthcare-oriented life-critical systems. The current Food and Drug Administration (FDA) regulatory requirements and guidance documentation do not address certain aspects of complete software assurance activities. In addition, the FDA's software oversight processes require enhancement to include increasingly complex healthcare systems such as Hospital Information Systems (HIS). The importance of complete software assurance is introduced, current regulatory requirements and guidance discussed, and the necessity for enhancements to the current processes shall be highlighted.

Some unintended consequences of information technology in health care: the nature of patient care information system-related errors

Medical error reduction is an international issue, as is the implementation of patient care information systems (PCISs) as a potential means to achieving it. As researchers conducting separate studies in the United States, The Netherlands, and Australia, using similar qualitative methods to investigate implementing PCISs, the authors have encountered many instances in which PCIS applications seem to foster errors rather than reduce their likelihood. The authors describe the kinds of silent errors they have witnessed and, from their different social science perspectives (information science, sociology, and cognitive science), they interpret the nature of these errors. The errors fall into two main categories: those in the process of entering and retrieving information, and those in the communication and coordination process that the PCIS is supposed to support. The authors believe that with a heightened awareness of these issues, informaticians can educate, design systems, implement, and conduct research in such a way that they might be able to avoid the unintended consequences of these subtle silent errors.

Nursing documentation time during implementation of an electronic medical record

OBJECTIVE

To determine, within the context of all nursing duties, the amount of time nurses spend on documentation during the implementation of an electronic medical record (EMR) on an intrapartum unit.

BACKGROUND

Increased documentation needs during EMR implementation may necessitate increased staffing requirements in an already labor-intensive and demanding environment.

METHODS

A work-sampling study was conducted over a 14-day study period, and 18 of 84 (21%) potential 4-hour observation periods were selected. During each period, a single observer made 120 observations and, on locating a specific nurse, immediately recorded that nurse's activity on a standardized and validated instrument. Categories of nursing activities included documentation, bedside care, bedside supportive care, nonbedside care, and nonpatient care.

RESULTS

A total of 2160 observations were made. The total percentage of nursing time spent for documentation was 15.8%, 10.6% on paper and 5.2% on the computer. The percentage of time spent on documentation was independently associated with day versus night shifts (19.2% vs 12.4%, respectively).

CONCLUSIONS

Despite charting concurrently on both paper and computer, the amount of time spent on documentation was not excessive, and was consistent with previous studies in which neither electronic nor "double charting" occurred.

The use of handheld computers in clinical trials

A recently completed, randomized, double-blind placebo-controlled clinical trial is presented in which Palm handheld computers were used as a substitute for normal paper-based patient diaries. In this nasal provocation study, a common antihistamine approved for the treatment of seasonal allergic rhinitis was tested against placebo for evidence of additional properties. In addition to their medical examinations, the 12 study volunteers rated subjective complaints in a diary program on 4 examination days, for a duration of 4.5 hours every 15 minutes at each visit. This resulted in 903 data sets consisting of five questions each, or 4515 data points total. In this study the use of handheld computers resulted in an increase in data quality and shortened the time needed to close the database. Moreover, the benefit of electronic reminders for protocol compliance is clearly demonstrated. Our findings support the results found in the literature we reviewed. For more than 16 years, mobile computers have been supporting the implementation of clinical trials. Our review of 27 articles out of more than 100 clinical trials in which mobile computers have been used elaborates on the advantages and problems of this technology. We give a comprehensive overview of the various technologies as used in different settings, and then discuss the methodology of using mobile devices in comparison to traditional methods, the considerations that need to be made and things to be avoided in order to conduct a successful clinical trial with mobile tools. We conclude that mobile devices are very useful in most cases, especially when design and software validation aspects have been taken into account.

Physician satisfaction with two order entry systems

OBJECTIVES

In the wake of the Institute of Medicine report, To Err Is Human: Building a Safer Health System (LT Kohn, JM Corrigan, MS Donaldson, eds; Washington, DC: National Academy Press, 1999), numerous advisory panels are advocating widespread implementation of physician order entry as a means to reduce errors and improve patient safety. Successful implementation of an order entry system requires that attention be given to the user interface. The authors assessed physician satisfaction with the user interface of two different order entry systems-a commercially available product, and the Department of Veterans Affairs Computerized Patient Record System (CPRS).

DESIGN AND MEASUREMENT

A standardized instrument for measuring user satisfaction with physician order entry systems was mailed to internal medicine and medicine-pediatrics house staff physicians. The subjects answered questions on each system using a 0 to 9 scale.

RESULTS

The survey response rates were 63 and 64 percent for the two order entry systems. Overall, house staff were dissatisfied with the commercial system, giving it an overall mean score of 3.67 (95 percent confidence interval [95%CI], 3.37-3.97). In contrast, the CPRS had a mean score of 7.21 (95% CI, 7.00-7.43), indicating that house staff were satisfied with the system. Overall satisfaction was most strongly correlated with the ability to perform tasks in a "straightforward" manner.

CONCLUSIONS

User satisfaction differed significantly between the two order entry systems, suggesting that all order entry systems are not equally usable. Given the national usage of the two order entry systems studied, further studies are needed to assess physician satisfaction with use of these same systems at other institutions.