1
|
Using Clinical Simulation to Evaluate AI-Enabled Decision Support. Stud Health Technol Inform 2024; 310:299-303. [PMID: 38269813 DOI: 10.3233/shti230975] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2024]
Abstract
Clinical simulation is a useful method for evaluating AI-enabled clinical decision support (CDS). Simulation studies permit patient- and risk-free evaluation and far greater experimental control than is possible with clinical studies. The effect of CDS assisted and unassisted patient scenarios on meaningful downstream decisions and actions within the information value chain can be evaluated as outcome measures. This paper discusses the use of clinical simulation in CDS evaluation and presents a case study to demonstrate feasibility of its application.
Collapse
|
2
|
How Well Do AI-Enabled Decision Support Systems Perform in Clinical Settings? Stud Health Technol Inform 2024; 310:279-283. [PMID: 38269809 DOI: 10.3233/shti230971] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2024]
Abstract
Real-world performance of machine learning (ML) models is crucial for safely and effectively embedding them into clinical decision support (CDS) systems. We examined evidence about the performance of contemporary ML-based CDS in clinical settings. A systematic search of four bibliographic databases identified 32 studies over a 5-year period. The CDS task, ML type, ML method and real-world performance was extracted and analysed. Most ML-based CDS supported image recognition and interpretation (n=12; 38%) and risk assessment (n=9; 28%). The majority used supervised learning (n=28; 88%) to train random forests (n=7; 22%) and convolutional neural networks (n=7; 22%). Only 12 studies reported real-world performance using heterogenous metrics; and performance degraded in clinical settings compared to model validation. The reporting of model performance is fundamental to ensuring safe and effective use of ML-based CDS in clinical settings. There remain opportunities to improve reporting.
Collapse
|
3
|
Automating the Identification of Safety Events Involving Machine Learning-Enabled Medical Devices. Stud Health Technol Inform 2024; 310:604-608. [PMID: 38269880 DOI: 10.3233/shti231036] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2024]
Abstract
With growing use of machine learning (ML)-enabled medical devices by clinicians and consumers safety events involving these systems are emerging. Current analysis of safety events heavily relies on retrospective review by experts, which is time consuming and cost ineffective. This study develops automated text classifiers and evaluates their potential to identify rare ML safety events from the US FDA's MAUDE. Four stratified classifiers were evaluated using a real-world data distribution with different feature sets: report text; text and device brand name; text and generic device type; and all information combined. We found that stratified classifiers using the generic type of devices were the most effective technique when tested on both stratified (F1-score=85%) and external datasets (precision=100%). All true positives on the external dataset were consistently identified by the three stratified classifiers, indicating the ensemble results from them can be used directly to monitor ML events reported to MAUDE.
Collapse
|
4
|
Effects of machine learning-based clinical decision support systems on decision-making, care delivery, and patient outcomes: a scoping review. J Am Med Inform Assoc 2023; 30:2050-2063. [PMID: 37647865 PMCID: PMC10654852 DOI: 10.1093/jamia/ocad180] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2023] [Revised: 08/01/2023] [Accepted: 08/23/2023] [Indexed: 09/01/2023] Open
Abstract
OBJECTIVE This study aims to summarize the research literature evaluating machine learning (ML)-based clinical decision support (CDS) systems in healthcare settings. MATERIALS AND METHODS We conducted a review in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta Analyses extension for Scoping Review). Four databases, including PubMed, Medline, Embase, and Scopus were searched for studies published from January 2016 to April 2021 evaluating the use of ML-based CDS in clinical settings. We extracted the study design, care setting, clinical task, CDS task, and ML method. The level of CDS autonomy was examined using a previously published 3-level classification based on the division of clinical tasks between the clinician and CDS; effects on decision-making, care delivery, and patient outcomes were summarized. RESULTS Thirty-two studies evaluating the use of ML-based CDS in clinical settings were identified. All were undertaken in developed countries and largely in secondary and tertiary care settings. The most common clinical tasks supported by ML-based CDS were image recognition and interpretation (n = 12) and risk assessment (n = 9). The majority of studies examined assistive CDS (n = 23) which required clinicians to confirm or approve CDS recommendations for risk assessment in sepsis and for interpreting cancerous lesions in colonoscopy. Effects on decision-making, care delivery, and patient outcomes were mixed. CONCLUSION ML-based CDS are being evaluated in many clinical areas. There remain many opportunities to apply and evaluate effects of ML-based CDS on decision-making, care delivery, and patient outcomes, particularly in resource-constrained settings.
Collapse
|
5
|
Artificial intelligence in medicine: has the time come to hang up the stethoscope? Intern Med J 2023; 53:1533-1539. [PMID: 37683094 PMCID: PMC10946904 DOI: 10.1111/imj.16216] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Accepted: 08/08/2023] [Indexed: 09/10/2023]
Abstract
The question of whether the time has come to hang up the stethoscope is bound up in the promises of artificial intelligence (AI), promises that have so far proven difficult to deliver, perhaps because of the mismatch between the technical capability of AI and its use in real-world clinical settings. This perspective argues that it is time to move away from discussing the generalised promise of disembodied AI and focus on specifics. We need to focus on how the computational method underlying AI, i.e. machine learning (ML), is embedded into tools, how those tools contribute to clinical tasks and decisions and to what extent they can be relied on. Accordingly, we pose four questions that must be asked to make the discussion real and to understand how ML tools contribute to health care: (1) What does the ML algorithm do? (2) How is output of the ML algorithm used in clinical tools? (3) What does the ML tool contribute to clinical tasks or decisions? (4) Can clinicians act or rely on the ML tool? Two exemplar ML tools are examined to show how these questions can be used to better understand the role of ML in supporting clinical tasks and decisions. Ultimately, ML is just a fancy method of automation. We show that it is useful in automating specific and narrowly defined clinical tasks but likely incapable of automating the full gamut of decisions and tasks performed by clinicians.
Collapse
|
6
|
Abstract
AIMS AND OBJECTIVES To examine the nature and use of automation in contemporary clinical information systems by reviewing studies reporting the implementation and evaluation of artificial intelligence (AI) technologies in healthcare settings. METHOD PubMed/MEDLINE, Web of Science, EMBASE, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies reporting evaluation of AI in clinical settings from January 2021 to December 2022. We documented the clinical application areas and tasks supported, and the level of system autonomy. Reported effects on user experience, decision-making, care delivery and outcomes were summarised. RESULTS AI technologies are being applied in a wide variety of clinical areas. Most contemporary systems utilise deep learning, use routinely collected data, support diagnosis and triage, are assistive (requiring users to confirm or approve AI provided information or decisions), and are used by doctors in acute care settings in high-income nations. AI systems are integrated and used within existing clinical information systems including electronic medical records. There is limited support for One Health goals. Evaluation is largely based on quantitative methods measuring effects on decision-making. CONCLUSION AI systems are being implemented and evaluated in many clinical areas. There remain many opportunities to understand patterns of routine use and evaluate effects on decision-making, care delivery and patient outcomes using mixed-methods. Support for One Health including integrating data about environmental factors and social determinants needs further exploration.
Collapse
|
7
|
More than algorithms: an analysis of safety events involving ML-enabled medical devices reported to the FDA. J Am Med Inform Assoc 2023; 30:1227-1236. [PMID: 37071804 PMCID: PMC10280342 DOI: 10.1093/jamia/ocad065] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2022] [Revised: 03/13/2023] [Accepted: 03/30/2023] [Indexed: 04/20/2023] Open
Abstract
OBJECTIVE To examine the real-world safety problems involving machine learning (ML)-enabled medical devices. MATERIALS AND METHODS We analyzed 266 safety events involving approved ML medical devices reported to the US FDA's MAUDE program between 2015 and October 2021. Events were reviewed against an existing framework for safety problems with Health IT to identify whether a reported problem was due to the ML device (device problem) or its use, and key contributors to the problem. Consequences of events were also classified. RESULTS Events described hazards with potential to harm (66%), actual harm (16%), consequences for healthcare delivery (9%), near misses that would have led to harm if not for intervention (4%), no harm or consequences (3%), and complaints (2%). While most events involved device problems (93%), use problems (7%) were 4 times more likely to harm (relative risk 4.2; 95% CI 2.5-7). Problems with data input to ML devices were the top contributor to events (82%). DISCUSSION Much of what is known about ML safety comes from case studies and the theoretical limitations of ML. We contribute a systematic analysis of ML safety problems captured as part of the FDA's routine post-market surveillance. Most problems involved devices and concerned the acquisition of data for processing by algorithms. However, problems with the use of devices were more likely to harm. CONCLUSIONS Safety problems with ML devices involve more than algorithms, highlighting the need for a whole-of-system approach to safe implementation with a special focus on how users interact with devices.
Collapse
|
8
|
Emergency department and urgent care clinician perspectives on digital access to past medical histories. BMJ Health Care Inform 2022; 29:bmjhci-2022-100567. [PMID: 35618316 PMCID: PMC9137332 DOI: 10.1136/bmjhci-2022-100567] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2022] [Accepted: 05/13/2022] [Indexed: 11/04/2022] Open
Abstract
ObjectiveTo explore emergency department (ED) and urgent care (UC) clinicians’ perceptions of digital access to patients’ past medical history (PMH).MethodsAn online survey compared anticipated and actual value of access to digital PMH. UTAUT2 (Unified Theory of Acceptance and Use of Technology 2) was used to assess technology acceptance. Quantitative data were analysed using Mann-Whitney U tests and qualitative data were analysed using a general inductive approach.Results33 responses were received. 94% (16/17) of respondents with PMH access said they valued their PMH system and all respondents with no digital PMH access (100%; 16/16) said they believed access would be valuable. Both groups indicated a high level of technology acceptance across all UTAUT2 dimensions. Free-text responses suggested improvements such as increasing the number of patient records available, standardisation of information presentation, increased system reliability, expanded access to information and validation by authoritative/trusted sources.DiscussionNon-PMH respondents’ expectations were closely matched with the benefits obtained by PMH respondents. High levels of technology acceptance indicated a strong willingness to adopt. Clinicians appeared clear about the improvements they would like for PMH content and access. Policy implications include the need to focus on higher levels of patient participation, and increasing the breadth and depth of information and processes to ensure patient record curation and stewardship.ConclusionThere appears to be strong clinician support for digital access to PMH in ED and UC; however, current systems appear to have many shortcomings.
Collapse
|
9
|
How Well Do Computerised Decision Support Systems Cover Nursing Standards of Practice? A Literature Review. Stud Health Technol Inform 2021; 284:269-274. [PMID: 34920524 DOI: 10.3233/shti210721] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
The use of computerized decision support systems (DSS) in nursing practice is increasing. However, research about who uses DSS, where are they implemented, and how they are linked with standards of nursing is limited. This paper presents evidence on users and settings of DSS implementation, along with specific nursing standards of practice that are facilitated by such DSS. We searched six bibliographic databases using relevant terms and identified 28 studies, each evaluating a unique DSS. Of these, 24 were used by registered nurses and 19 were implemented in short-term care units. Most of the DSS were found to facilitate nursing standards of assessment and intervention, however, outcome identification and evaluation were the least included standards. These findings not only highlight gaps in systems but also offer opportunities for further research development in this area.
Collapse
|
10
|
Automation in nursing decision support systems: A systematic review of effects on decision making, care delivery, and patient outcomes. J Am Med Inform Assoc 2021; 28:2502-2513. [PMID: 34498063 PMCID: PMC8510331 DOI: 10.1093/jamia/ocab123] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2021] [Revised: 05/24/2021] [Accepted: 06/03/2021] [Indexed: 01/07/2023] Open
Abstract
OBJECTIVE The study sought to summarize research literature on nursing decision support systems (DSSs ); understand which steps of the nursing care process (NCP) are supported by DSSs, and analyze effects of automated information processing on decision making, care delivery, and patient outcomes. MATERIALS AND METHODS We conducted a systematic review in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement. PubMed, CINAHL, Cochrane, Embase, Scopus, and Web of Science were searched from January 2014 to April 2020 for studies focusing on DSSs used exclusively by nurses and their effects. Information about the stages of automation (information acquisition, information analysis, decision and action selection, and action implementation), NCP, and effects was assessed. RESULTS Of 1019 articles retrieved, 28 met the inclusion criteria, each studying a unique DSS. Most DSSs were concerned with two NCP steps: assessment (82%) and intervention (86%). In terms of automation, all included DSSs automated information analysis and decision selection. Five DSSs automated information acquisition and only one automated action implementation. Effects on decision making, care delivery, and patient outcome were mixed. DSSs improved compliance with recommendations and reduced decision time, but impacts were not always sustainable. There were improvements in evidence-based practice, but impact on patient outcomes was mixed. CONCLUSIONS Current nursing DSSs do not adequately support the NCP and have limited automation. There remain many opportunities to enhance automation, especially at the stage of information acquisition. Further research is needed to understand how automation within the NCP can improve nurses' decision making, care delivery, and patient outcomes.
Collapse
|
11
|
How machine learning is embedded to support clinician decision making: an analysis of FDA-approved medical devices. BMJ Health Care Inform 2021; 28:bmjhci-2020-100301. [PMID: 33853863 PMCID: PMC8054073 DOI: 10.1136/bmjhci-2020-100301] [Citation(s) in RCA: 28] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2020] [Revised: 02/10/2021] [Accepted: 03/02/2021] [Indexed: 12/20/2022] Open
Abstract
Objective To examine how and to what extent medical devices using machine learning (ML) support clinician decision making. Methods We searched for medical devices that were (1) approved by the US Food and Drug Administration (FDA) up till February 2020; (2) intended for use by clinicians; (3) in clinical tasks or decisions and (4) used ML. Descriptive information about the clinical task, device task, device input and output, and ML method were extracted. The stage of human information processing automated by ML-based devices and level of autonomy were assessed. Results Of 137 candidates, 59 FDA approvals for 49 unique devices were included. Most approvals (n=51) were since 2018. Devices commonly assisted with diagnostic (n=35) and triage (n=10) tasks. Twenty-three devices were assistive, providing decision support but left clinicians to make important decisions including diagnosis. Twelve automated the provision of information (autonomous information), such as quantification of heart ejection fraction, while 14 automatically provided task decisions like triaging the reading of scans according to suspected findings of stroke (autonomous decisions). Stages of human information processing most automated by devices were information analysis, (n=14) providing information as an input into clinician decision making, and decision selection (n=29), where devices provide a decision. Conclusion Leveraging the benefits of ML algorithms to support clinicians while mitigating risks, requires a solid relationship between clinician and ML-based devices. Such relationships must be carefully designed, considering how algorithms are embedded in devices, the tasks supported, information provided and clinicians’ interactions with them.
Collapse
|
12
|
Health disparities among pregnant women with sars-cov-2 infection at a university medical center in northern California. Am J Obstet Gynecol 2020. [PMCID: PMC7683952 DOI: 10.1016/j.ajog.2020.08.138] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
|
13
|
Emergency care access to primary care records: an observational study. BMJ Health Care Inform 2020; 27:bmjhci-2020-100153. [PMID: 32830108 PMCID: PMC7445344 DOI: 10.1136/bmjhci-2020-100153] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2020] [Revised: 06/18/2020] [Accepted: 07/08/2020] [Indexed: 12/23/2022] Open
Abstract
OBJECTIVE To measure lookup rates of externally held primary care records accessed in emergency care and identify patient characteristics, conditions and potential consequences associated with access. MEASURES Rates of primary care record access and re-presentation to the emergency department (ED) within 30 days and hospital admission. DESIGN A retrospective observational study of 77 181 ED presentations over 4 years and 9 months, analysing 8184 index presentations in which patients' primary care records were accessed from the ED. Data were compared with 17 449 randomly selected index control presentations. Analysis included propensity score matching for age and triage categories. RESULTS 6.3% of overall ED presentations triggered a lookup (rising to 8.3% in year 5); 83.1% of patients were only looked up once and 16.9% of patients looked up on multiple occasions. Lookup patients were on average 25 years older (z=-9.180, p<0.001, r=0.43). Patients with more urgent triage classifications had their records accessed more frequently (z=-36.47, p<0.001, r=0.23). Record access was associated with a significant but negligible increase in hospital admission (χ2 (1, n=13 120)=98.385, p<0.001, phi=0.087) and readmission within 30 days (χ2 (1, n=13 120)=86.288, p<0.001, phi=0.081). DISCUSSION Emergency care clinicians access primary care records more frequently for older patients or those in higher triage categories. Increased levels of inpatient admission and re-presentation within 30 days are likely linked to age and triage categories. CONCLUSION Further studies should focus on the impact of record access on clinical and process outcomes and which record elements have the most utility to shape clinical decisions.
Collapse
|
14
|
Effect of Speech Recognition on Problem Solving and Recall in Consumer Digital Health Tasks: Controlled Laboratory Experiment. J Med Internet Res 2020; 22:e14827. [PMID: 32442129 PMCID: PMC7296411 DOI: 10.2196/14827] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2019] [Revised: 11/27/2019] [Accepted: 03/29/2020] [Indexed: 11/22/2022] Open
Abstract
Background Recent advances in natural language processing and artificial intelligence have led to widespread adoption of speech recognition technologies. In consumer health applications, speech recognition is usually applied to support interactions with conversational agents for data collection, decision support, and patient monitoring. However, little is known about the use of speech recognition in consumer health applications and few studies have evaluated the efficacy of conversational agents in the hands of consumers. In other consumer-facing tools, cognitive load has been observed to be an important factor affecting the use of speech recognition technologies in tasks involving problem solving and recall. Users find it more difficult to think and speak at the same time when compared to typing, pointing, and clicking. However, the effects of speech recognition on cognitive load when performing health tasks has not yet been explored. Objective The aim of this study was to evaluate the use of speech recognition for documentation in consumer digital health tasks involving problem solving and recall. Methods Fifty university staff and students were recruited to undertake four documentation tasks with a simulated conversational agent in a computer laboratory. The tasks varied in complexity determined by the amount of problem solving and recall required (simple and complex) and the input modality (speech recognition vs keyboard and mouse). Cognitive load, task completion time, error rate, and usability were measured. Results Compared to using a keyboard and mouse, speech recognition significantly increased the cognitive load for complex tasks (Z=–4.08, P<.001) and simple tasks (Z=–2.24, P=.03). Complex tasks took significantly longer to complete (Z=–2.52, P=.01) and speech recognition was found to be overall less usable than a keyboard and mouse (Z=–3.30, P=.001). However, there was no effect on errors. Conclusions Use of a keyboard and mouse was preferable to speech recognition for complex tasks involving problem solving and recall. Further studies using a broader variety of consumer digital health tasks of varying complexity are needed to investigate the contexts in which use of speech recognition is most appropriate. The effects of cognitive load on task performance and its significance also need to be investigated.
Collapse
|
15
|
Health resource utilization of labor induction versus expectant management. Am J Obstet Gynecol 2020; 222:369.e1-369.e11. [PMID: 31930993 DOI: 10.1016/j.ajog.2020.01.002] [Citation(s) in RCA: 20] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2019] [Revised: 11/26/2019] [Accepted: 01/06/2020] [Indexed: 11/25/2022]
Abstract
BACKGROUND Although induction of labor of low-risk nulliparous women at 39 weeks reduces the risk of cesarean delivery compared with expectant management, concern regarding more frequent use of labor induction remains, given that this intervention historically has been thought to incur greater resource utilization. OBJECTIVE The objective of the study was to determine whether planned elective labor induction at 39 weeks among low-risk nulliparous women, compared with expectant management, was associated with differences in health care resource utilization from the time of randomization through 8 weeks postpartum. STUDY DESIGN This is a planned secondary analysis of a multicenter randomized trial in which low-risk nulliparous women were assigned to induction of labor at 39 weeks or expectant management. We assessed resource utilization after randomization in 3 time periods: antepartum, delivery admission, and discharge through 8 weeks postpartum. RESULTS Of 6096 women with data available, those in the induction of labor group (n = 3059) were significantly less likely in the antepartum period after randomization to have at least 1 ambulatory visit for routine prenatal care (32.4% vs 68.4%), unanticipated care (0.5% vs 2.6%), or urgent care (16.2% vs 44.3%), or at least 1 antepartum hospitalization (0.8% vs 2.2%, P < .001 for all). They also had fewer tests (eg, sonograms, blood tests) and treatments (eg, antibiotics, intravenous hydration) prior to delivery. During the delivery admission, women in the induction of labor group spent a longer time in labor and delivery (median, 0.83 vs 0.57 days), but both women (P = .002) and their neonates (P < .001) had shorter postpartum stays. Women and neonates in both groups had similar frequencies of postpartum urgent care and hospital readmissions (P > .05 for all). CONCLUSION Women randomized to induction of labor had longer durations in labor and delivery but significantly fewer antepartum visits, tests, and treatments and shorter maternal and neonatal hospital durations after delivery. These results demonstrate that the health outcome advantages associated with induction of labor are gained without incurring uniformly greater health care resource use.
Collapse
|
16
|
Abstract
OBJECTIVE Clinicians using clinical decision support (CDS) to prescribe medications have an obligation to ensure that prescriptions are safe. One option is to verify the safety of prescriptions if there is uncertainty, for example, by using drug references. Supervisory control experiments in aviation and process control have associated errors, with reduced verification arising from overreliance on decision support. However, it is unknown whether this relationship extends to clinical decision-making. Therefore, we examine whether there is a relationship between verification behaviors and prescribing errors, with and without CDS medication alerts, and whether task complexity mediates this. METHODS A total of 120 students in the final 2 years of a medical degree prescribed medicines for patient scenarios using a simulated electronic prescribing system. CDS (correct, incorrect, and no CDS) and task complexity (low and high) were varied. Outcomes were omission (missed prescribing errors) and commission errors (accepted false-positive alerts). Verification measures were access of drug references and view time percentage of task time. RESULTS Failure to access references for medicines with prescribing errors increased omission errors with no CDS (high-complexity: χ 2(1) = 12.716; p < 0.001) and incorrect CDS (Fisher's exact; low-complexity: p = 0.002; high-complexity: p = 0.001). Failure to access references for false-positive alerts increased commission errors (low-complexity: χ 2(1) = 16.673, p < 0.001; high-complexity: χ 2(1) = 18.690, p < 0.001). Fewer participants accessed relevant references with incorrect CDS compared with no CDS (McNemar; low-complexity: p < 0.001; high-complexity: p < 0.001). Lower view time percentages increased omission (F(3, 361.914) = 4.498; p = 0.035) and commission errors (F(1, 346.223) = 2.712; p = 0.045). View time percentages were lower in CDS-assisted conditions compared with unassisted conditions (F(2, 335.743) = 10.443; p < 0.001). DISCUSSION The presence of CDS reduced verification of prescription safety. When CDS was incorrect, reduced verification was associated with increased prescribing errors. CONCLUSION CDS can be incorrect, and verification provides one mechanism to detect errors. System designers need to facilitate verification without increasing workload or eliminating the benefits of correct CDS.
Collapse
|
17
|
The Effect of Cognitive Load and Task Complexity on Automation Bias in Electronic Prescribing. HUMAN FACTORS 2018; 60:1008-1021. [PMID: 29939764 DOI: 10.1177/0018720818781224] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
OBJECTIVE Determine the relationship between cognitive load (CL) and automation bias (AB). BACKGROUND Clinical decision support (CDS) for electronic prescribing can improve safety but introduces the risk of AB, where reliance on CDS replaces vigilance in information seeking and processing. We hypothesized high CL generated by high task complexity would increase AB errors. METHOD One hundred twenty medical students prescribed medicines for clinical scenarios using a simulated e-prescribing system in a randomized controlled experiment. Quality of CDS (correct, incorrect, and no CDS) and task complexity (low and high) were varied. CL, omission errors (failure to detect prescribing errors), and commission errors (acceptance of false positive alerts) were measured. RESULTS Increasing complexity from low to high significantly increased CL, F(1, 118) = 71.6, p < .001. CDS reduced CL in high-complexity conditions compared to no CDS, F(2, 117) = 4.72, p = .015. Participants who made omission errors in incorrect and no CDS conditions exhibited lower CL compared to those who did not, F(1, 636.49) = 3.79, p = .023. CONCLUSION Results challenge the notion that AB is triggered by increasing task complexity and associated increases in CL. Omission errors were associated with lower CL, suggesting errors may stem from an insufficient allocation of cognitive resources. APPLICATION This is the first research to examine the relationship between CL and AB. Findings suggest designers and users of CDS systems need to be aware of the risks of AB. Interventions that increase user vigilance and engagement may be beneficial and deserve further investigation.
Collapse
|
18
|
Early versus late maternal readmission for sepsis. Am J Obstet Gynecol 2017. [DOI: 10.1016/j.ajog.2017.08.033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
19
|
Risk factors for maternal readmission with sepsis. Am J Obstet Gynecol 2017. [DOI: 10.1016/j.ajog.2017.08.032] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
20
|
Abstract
Background Clinical decision support (CDS) in e-prescribing can improve safety by alerting potential errors, but introduces new sources of risk. Automation bias (AB) occurs when users over-rely on CDS, reducing vigilance in information seeking and processing. Evidence of AB has been found in other clinical tasks, but has not yet been tested with e-prescribing. This study tests for the presence of AB in e-prescribing and the impact of task complexity and interruptions on AB. Methods One hundred and twenty students in the final two years of a medical degree prescribed medicines for nine clinical scenarios using a simulated e-prescribing system. Quality of CDS (correct, incorrect and no CDS) and task complexity (low, low + interruption and high) were varied between conditions. Omission errors (failure to detect prescribing errors) and commission errors (acceptance of false positive alerts) were measured. Results Compared to scenarios with no CDS, correct CDS reduced omission errors by 38.3% (p < .0001, n = 120), 46.6% (p < .0001, n = 70), and 39.2% (p < .0001, n = 120) for low, low + interrupt and high complexity scenarios respectively. Incorrect CDS increased omission errors by 33.3% (p < .0001, n = 120), 24.5% (p < .009, n = 82), and 26.7% (p < .0001, n = 120). Participants made commission errors, 65.8% (p < .0001, n = 120), 53.5% (p < .0001, n = 82), and 51.7% (p < .0001, n = 120). Task complexity and interruptions had no impact on AB. Conclusions This study found evidence of AB omission and commission errors in e-prescribing. Verification of CDS alerts is key to avoiding AB errors. However, interventions focused on this have had limited success to date. Clinicians should remain vigilant to the risks of CDS failures and verify CDS. Electronic supplementary material The online version of this article (doi:10.1186/s12911-017-0425-5) contains supplementary material, which is available to authorized users.
Collapse
|
21
|
Automation bias and verification complexity: a systematic review. J Am Med Inform Assoc 2017; 24:423-431. [PMID: 27516495 PMCID: PMC7651899 DOI: 10.1093/jamia/ocw105] [Citation(s) in RCA: 82] [Impact Index Per Article: 11.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2016] [Revised: 05/24/2016] [Accepted: 05/27/2016] [Indexed: 01/01/2023] Open
Abstract
INTRODUCTION While potentially reducing decision errors, decision support systems can introduce new types of errors. Automation bias (AB) happens when users become overreliant on decision support, which reduces vigilance in information seeking and processing. Most research originates from the human factors literature, where the prevailing view is that AB occurs only in multitasking environments. OBJECTIVES This review seeks to compare the human factors and health care literature, focusing on the apparent association of AB with multitasking and task complexity. DATA SOURCES EMBASE, Medline, Compendex, Inspec, IEEE Xplore, Scopus, Web of Science, PsycINFO, and Business Source Premiere from 1983 to 2015. STUDY SELECTION Evaluation studies where task execution was assisted by automation and resulted in errors were included. Participants needed to be able to verify automation correctness and perform the task manually. METHODS Tasks were identified and grouped. Task and automation type and presence of multitasking were noted. Each task was rated for its verification complexity. RESULTS Of 890 papers identified, 40 met the inclusion criteria; 6 were in health care. Contrary to the prevailing human factors view, AB was found in single tasks, typically involving diagnosis rather than monitoring, and with high verification complexity. LIMITATIONS The literature is fragmented, with large discrepancies in how AB is reported. Few studies reported the statistical significance of AB compared to a control condition. CONCLUSION AB appears to be associated with the degree of cognitive load experienced in decision tasks, and appears to not be uniquely associated with multitasking. Strategies to minimize AB might focus on cognitive load reduction.
Collapse
|
22
|
Anticoagulant prescribing practices and anesthetic interventions among anticoagulated pregnant patients: a retrospective study. Int J Obstet Anesth 2014; 23:238-45. [PMID: 24910353 DOI: 10.1016/j.ijoa.2014.03.007] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/30/2013] [Revised: 02/27/2014] [Accepted: 03/16/2014] [Indexed: 10/25/2022]
Abstract
BACKGROUND The peripartum management of anticoagulated patients poses important challenges for obstetric anesthesiologists, especially when deciding to perform neuraxial block. However, there is limited evidence evaluating anticoagulant prescribing practices and neuraxial block utilization in this setting. Our objective was to examine peripartum anticoagulant prescribing and anesthetic practices in a cohort of anticoagulated patients receiving subcutaneous enoxaparin, and subcutaneous or intravenous unfractionated heparin. METHODS We performed a retrospective study of anticoagulant prescribing patterns and anesthetic interventions among patients receiving enoxaparin and/or unfractionated heparin who delivered at a USA obstetric center over a seven-year period. RESULTS We identified 101 patients who received enoxaparin and/or unfractionated heparin before delivery. Thirty-nine (38.6%) patients received enoxaparin only, 41 (40.6%) patients received enoxaparin bridged to subcutaneous unfractionated heparin, 11 (10.9%) patients received enoxaparin and were converted to intravenous unfractionated heparin and 10 (9.9%) patients received only subcutaneous unfractionated heparin. The overall rate of neuraxial block utilization was 80.2%. The median [IQR] time-period between the last dose of enoxaparin and neuraxial block was significantly shorter for patients who received only enoxaparin vs. enoxaparin with subcutaneous unfractionated heparin (54h [12-192h] (n=26) vs. 216h [39-504h] (n=23), P=0.04). Among patients who received only subcutaneous unfractionated heparin, the time period between unfractionated heparin and neuraxial block was 24h [19-51h]. CONCLUSION In this study, the neuraxial block rate was high among patients receiving enoxaparin and/or subcutaneous unfractionated heparin with patients undergoing relatively long periods off anticoagulation. Careful multidisciplinary planning is recommended for the peripartum management of anticoagulated pregnant patients.
Collapse
|
23
|
Neonatal chest wall rigidity following the use of remifentanil for cesarean delivery in a patient with autoimmune hepatitis and thrombocytopenia. Int J Obstet Anesth 2004; 13:53-6. [PMID: 15321443 DOI: 10.1016/j.ijoa.2003.09.001] [Citation(s) in RCA: 30] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 09/01/2003] [Indexed: 11/29/2022]
Abstract
Remifentanil is a useful adjunct in general anesthesia for high-risk obstetric patients. It provides effective blunting of the rapid hemodynamic changes that may be associated with airway manipulation and surgical stimulation. There have been no previous reports of opioid-related rigidity in the neonate delivered by a parturient receiving intraoperative remifentanil. We present a case of short-lived neonatal rigidity and respiratory depression following remifentanil administration during cesarean section to a parturient with autoimmune hepatitis complicated by cirrhosis, esophageal varices and thrombocytopenia.
Collapse
|