Learning from errors: analysis of medication order voiding in CPOE systems

The use of artificial intelligence to detect voided medication orders in oral and maxillofacial surgery inpatients

OBJECTIVE

The aim of this study is to determine if supervised machine learning algorithms can accurately predict voided computerized physician order entry in oral and maxillofacial surgery inpatients.

METHODS

Data from Electronic Medical Record included patient demographics, comorbidities, procedures, vital signs, laboratory values, and medication orders were retrospectively collected. Predictor variables included patient demographics, comorbidities, procedures, vital signs, and laboratory values. Outcome of interest is if a medication order was voided or not. Data was cleaned and processed using Microsoft Excel and Python v3.12. Gradient Boosted Decision Trees, Random Forest, K-Nearest Neighbor, and Naïve Bayes were trained, validated, and tested for accuracy of the prediction of voided medication orders.

RESULTS

37,493 medication orders from 1,204 patient admissions over 5 years were used for this study. 3,892 (10.4%) medication orders were voided. Gradient Boosted Decision Trees, Random Forest, K-Nearest Neighbor, and Naïve Bayes had an Area Under the Receiver Operating Curve of 0.802 with 95% CI [0.787, 0.825], 0.746 with 95% CI [0.722, 0.765], 0.685 with 95% CI [0.667, 0.699], and 0.505 with 95% CI [0.489, 0.539], respectively. Area Under the Precision Recall Curve was 0.684 with 95% CI [0.679, 0.702], 0.647 with 95% CI [0.638, 0.664], 0.429 with 95% CI [0.417, 0.434], and 0.551 with 95% CI [0.551, 0.552], respectively.

CONCLUSION

Gradient Boosted Decision Trees was the best performing model of the supervised machine learning algorithms with satisfactory outcomes in the test cohort for predicting voided Computerized Physician Order Entry in Oral and Maxillofacial Surgery inpatients.

Automated search methods for identifying wrong patient order entry-a scoping review

Objective

To investigate: (1) what automated search methods are used to identify wrong-patient order entry (WPOE), (2) what data are being captured and how they are being used, (3) the causes of WPOE, and (4) how providers identify their own errors.

Materials and Methods

A systematic scoping review of the empirical literature was performed using the databases CINAHL, Embase, and MEDLINE, covering the period from database inception until 2021. Search terms were related to the use of automated searches for WPOE when using an electronic prescribing system. Data were extracted and thematic analysis was performed to identify patterns or themes within the data.

Results

Fifteen papers were included in the review. Several automated search methods were identified, with the retract-and-reorder (RAR) method and the Void Alert Tool (VAT) the most prevalent. Included studies used automated search methods to identify background error rates in isolation, or in the context of an intervention. Risk factors for WPOE were identified, with technological factors and interruptions deemed the biggest risks. Minimal data on how providers identify their own errors were identified.

Discussion

RAR is the most widely used method to identify WPOE, with a good positive predictive value (PPV) of 76.2%. However, it will not currently identify other error types. The VAT is nonspecific for WPOE, with a mean PPV of 78%-93.1%, but the voiding reason accuracy varies considerably.

Conclusion

Automated search methods are powerful tools to identify WPOE that would otherwise go unnoticed. Further research is required around self-identification of errors.

Identification of Prescribing Errors in an Electronic Health Record Using a Retract-and-Reorder Tool: A Pilot Study

OBJECTIVES

The aims of this study were to develop and to validate an adapted Retract-and-Reorder (RAR) tool to identify and quantify near-miss/intercepted prescribing errors in an electronic health record.

METHODS

This is a cross-sectional study between February and March 2021 in an Irish maternity hospital. We used the RAR tool to detect near-miss prescribing errors in audit log data. Potential errors flagged by the tool were validated using prescriber interviews. Chart reviews were performed if the prescriber was unavailable for interview. Errors were judged to be clinical decisions in chart reviews through review of narrative notes, order components, and patient's clinical history. Interviews were analyzed with reference to the London Protocol, a process of incident analysis that categorizes causes of errors into various contributory factors including patient factors, task and technology factors, and work environment. Logistic regression with robust clustered standard errors was used to determine predictors for near-miss prescribing errors. We calculated the positive predictive value of the RAR tool by dividing the number of confirmed near-miss prescribing errors by the total number of RAR events identified.

RESULTS

Eighty-four RAR events were identified in 27,407 medication orders. Seventy-one events were confirmed near-miss prescribing errors, resulting in a positive predictive value of 85.0% (95% confidence interval, 75%-91%) and an estimated near-miss prescribing error rate of 259/100,000 medication orders. Duplicate prescribing errors were most common (54/71, 76.1%). No errors were reported by prescribers. Consultants were less likely to make an error than nonconsultant hospital doctors (adjusted odds ratio, 0.10; 95% confidence interval, 0.01-0.84). Factors associated with errors included workload, staffing levels, and task structure.

CONCLUSIONS

Our adapted RAR tool identified a variety of near-miss prescribing errors not otherwise reported. The tool has been implemented in the study hospital as a patient safety resource. Further implementations are planned across Irish hospitals.

The impact of clinical decision support systems on provider behavior in the inpatient setting: A systematic review and meta-analysis

BACKGROUND

Clinical decision support systems (CDSS) are used to improve processes of care. CDSS proliferation may have unintended consequences impacting effectiveness.

OBJECTIVE

To evaluate the effectiveness of CDSS in altering clinician behavior.

DESIGN

Electronic searches were performed in EMBASE, PubMed, and Cochrane Central Register of Control Trials for randomized controlled trials testing the impacted of CDSS on clinician behavior from 2000-2021. Extracted data included study design, CDSS attributed and outcomes, user characteristics, settings, and risk of bias. Eligible studies were analyzed qualitatively to describe CDSS types. Studies with sufficient outcome data were included in the meta-analysis.

SETTING AND PARTICIPANTS

Adult inpatients in the United States.

INTERVENTION

Clinical decision support system versus non-clinical decision support system.

MAIN OUTCOME AND MEASURE

A random-effects model measured the pooled risk difference (RD) and odds ratio of clinicians' adherence to CDSS; subgroup analyses tested differences in CDSS effectiveness over time and by CDSS type.

RESULTS

Qualitative synthesis included 22 studies. Eleven studies reported sufficient outcome data for inclusion in the meta-analysis. CDSS did not result in a statistically significant increase in clinician adoption of desired practicies (RD = 0.04 [95% confidence interval {CI} 0.00, 0.07]). CDSS from 2010-2015 (n = 5) did not increase clinician adoption of desired practice [RD -0.01, (95% CI -0.04, 0.02)].CDSS from 2016-2021 (n = 6) were associated with an increase in targeted practices [RD 0.07 (95% CI0.03, 0.12)], pInteraction = 0.004. EHR [RD 0.04 (95% CI 0.00, 0.08)] vs. non-EHR [RD 0.01 (95% CI -0.01, 0.04)] based CDSS interventions did not result in different adoption of desired practices (pInteraction = 0.27). The meta-analysis did not find an overall positive impact of CDSS on clinician behavior in the inpatient setting.

Predicting self-intercepted medication ordering errors using machine learning

Current approaches to understanding medication ordering errors rely on relatively small manually captured error samples. These approaches are resource-intensive, do not scale for computerized provider order entry (CPOE) systems, and are likely to miss important risk factors associated with medication ordering errors. Previously, we described a dataset of CPOE-based medication voiding accompanied by univariable and multivariable regression analyses. However, these traditional techniques require expert guidance and may perform poorly compared to newer approaches. In this paper, we update that analysis using machine learning (ML) models to predict erroneous medication orders and identify its contributing factors. We retrieved patient demographics (race/ethnicity, sex, age), clinician characteristics, type of medication order (inpatient, prescription, home medication by history), and order content. We compared logistic regression, random forest, boosted decision trees, and artificial neural network models. Model performance was evaluated using area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). The dataset included 5,804,192 medication orders, of which 28,695 (0.5%) were voided. ML correctly classified voids at reasonable accuracy; with a positive predictive value of 10%, ~20% of errors were included. Gradient boosted decision trees achieved the highest AUROC (0.7968) and AUPRC (0.0647) among all models. Logistic regression had the poorest performance. Models identified predictive factors with high face validity (e.g., student orders), and a decision tree revealed interacting contexts with high rates of errors not identified by previous regression models. Prediction models using order-entry information offers promise for error surveillance, patient safety improvements, and targeted clinical review. The improved performance of models with complex interactions points to the importance of contextual medication ordering information for understanding contributors to medication errors.

Risk factors associated with medication ordering errors

OBJECTIVE

We utilized a computerized order entry system-integrated function referred to as "void" to identify erroneous orders (ie, a "void" order). Using voided orders, we aimed to (1) identify the nature and characteristics of medication ordering errors, (2) investigate the risk factors associated with medication ordering errors, and (3) explore potential strategies to mitigate these risk factors.

MATERIALS AND METHODS

We collected data on voided orders using clinician interviews and surveys within 24 hours of the voided order and using chart reviews. Interviews were informed by the human factors-based SEIPS (Systems Engineering Initiative for Patient Safety) model to characterize the work systems-based risk factors contributing to ordering errors; chart reviews were used to establish whether a voided order was a true medication ordering error and ascertain its impact on patient safety.

RESULTS

During the 16-month study period (August 25, 2017, to December 31, 2018), 1074 medication orders were voided; 842 voided orders were true medication errors (positive predictive value = 78.3 ± 1.2%). A total of 22% (n = 190) of the medication ordering errors reached the patient, with at least a single administration, without causing patient harm. Interviews were conducted on 355 voided orders (33% response). Errors were not uniquely associated with a single risk factor, but the causal contributors of medication ordering errors were multifactorial, arising from a combination of technological-, cognitive-, environmental-, social-, and organizational-level factors.

CONCLUSIONS

The void function offers a practical, standardized method to create a rich database of medication ordering errors. We highlight implications for utilizing the void function for future research, practice and learning opportunities.

Conceptual considerations for using EHR-based activity logs to measure clinician burnout and its effects

Electronic health records (EHR) use is often considered a significant contributor to clinician burnout. Informatics researchers often measure clinical workload using EHR-derived audit logs and use it for quantifying the contribution of EHR use to clinician burnout. However, translating clinician workload measured using EHR-based audit logs into a meaningful burnout metric requires an alignment with the conceptual and theoretical principles of burnout. In this perspective, we describe a systems-oriented conceptual framework to achieve such an alignment and describe the pragmatic realization of this conceptual framework using 3 key dimensions: standardizing the measurement of EHR-based clinical work activities, implementing complementary measurements, and using appropriate instruments to assess burnout and its downstream outcomes. We discuss how careful considerations of such dimensions can help in augmenting EHR-based audit logs to measure factors that contribute to burnout and for meaningfully assessing downstream patient safety outcomes.

Effects of CPOE-based medication ordering on outcomes: an overview of systematic reviews

BACKGROUND

Computerised provider order entry (CPOE) systems are widely used in clinical settings for the electronic ordering of medications, laboratory tests and radiological therapies. However, evidence regarding effects of CPOE-based medication ordering on clinical and safety outcomes is mixed. We conducted an overview of systematic reviews (SRs) to characterise the cumulative effects of CPOE use for medication ordering in clinical settings.

METHODS

MEDLINE, EMBASE, CINAHL and the Cochrane Library were searched to identify published SRs from inception to 12 February 2018. SRs investigating the effects of the use of CPOE for medication ordering were included. Two reviewers independently extracted data and assessed the methodological quality of included SRs.

RESULTS

Seven SRs covering 118 primary studies were included for review. Pooled studies from the SRs in inpatient settings showed that CPOE use resulted in statistically significant decreases in medication errors and adverse drug events (ADEs); however, there was considerable variation in the magnitude of their relative risk reduction (54%-92% for errors, 35%-53% for ADEs). There was no significant relative risk reduction on hospital mortality or length of stay. Bibliographic analysis showed limited overlap (24%) among studies included across all SRs.

CONCLUSION

SRs on CPOEs included predominantly non-randomised controlled trials and observational studies with varying foci. SRs predominantly focused on inpatient settings and often lacked comparison groups; SRs used inconsistent definitions of outcomes, lacked descriptions regarding the effects on patient harm and did not differentiate among the levels of available decision support. With five of the seven SRs having low to moderate quality, findings from the SRs must be interpreted with caution. We discuss potential directions for future primary studies and SRs of CPOE.

Automated detection of wrong-drug prescribing errors

BACKGROUND

To assess the specificity of an algorithm designed to detect look-alike/sound-alike (LASA) medication prescribing errors in electronic health record (EHR) data.

SETTING

Urban, academic medical centre, comprising a 495-bed hospital and outpatient clinic running on the Cerner EHR. We extracted 8 years of medication orders and diagnostic claims. We licensed a database of medication indications, refined it and merged it with the medication data. We developed an algorithm that triggered for LASA errors based on name similarity, the frequency with which a patient received a medication and whether the medication was justified by a diagnostic claim. We stratified triggers by similarity. Two clinicians reviewed a sample of charts for the presence of a true error, with disagreements resolved by a third reviewer. We computed specificity, positive predictive value (PPV) and yield.

RESULTS

The algorithm analysed 488 481 orders and generated 2404 triggers (0.5% rate). Clinicians reviewed 506 cases and confirmed the presence of 61 errors, for an overall PPV of 12.1% (95% CI 10.7% to 13.5%). It was not possible to measure sensitivity or the false-negative rate. The specificity of the algorithm varied as a function of name similarity and whether the intended and dispensed drugs shared the same route of administration.

CONCLUSION

Automated detection of LASA medication errors is feasible and can reveal errors not currently detected by other means. Real-time error detection is not possible with the current system, the main barrier being the real-time availability of accurate diagnostic information. Further development should replicate this analysis in other health systems and on a larger set of medications and should decrease clinician time spent reviewing false-positive triggers by increasing specificity.

Genomics and electronic health record systems

Several reviews and case reports have described how information derived from the analysis of genomes are currently included in electronic health records (EHRs) for the purposes of supporting clinical decisions. Since the introduction of this new type of information in EHRs is relatively new (for instance, the widespread adoption of EHRs in the United States is just about a decade old), it is not surprising that a myriad of approaches has been attempted, with various degrees of success. EHR systems undergo much customization to fit the needs of health systems; these approaches have been varied and not always generalizable. The intent of this article is to present a high-level view of these approaches, emphasizing the functionality that they are trying to achieve, and not to advocate for specific solutions, which may become obsolete soon after this review is published. We start by broadly defining the end goal of including genomics in EHRs for healthcare and then explaining the various sources of information that need to be linked to arrive at a clinically actionable genomics analysis using a pharmacogenomics example. In addition, we include discussions on open issues and a vision for the next generation systems that integrate whole genome sequencing and EHRs in a seamless fashion.

Contributions from the 2017 Literature on Clinical Decision Support

Objectives:  To summarize recent research and select the best papers published in 2017 in the field of computerized clinical decision support for the Decision Support section of the International Medical Informatics Association (IMIA) yearbook.

Methods:  A literature review was performed by searching two bibliographic databases for papers referring to clinical decision support systems (CDSSs). The aim was to identify a list of candidate best papers from the retrieved bibliographic records, which were then peer-reviewed by external reviewers. A consensus meeting of the IMIA editorial team finally selected the best papers on the basis of all reviews and the section editors' evaluation.

Results:  Among the 1,194 retrieved papers, the entire review process resulted in the selection of four best papers. The first paper studies the impact of recency and of longitudinal extent of electronic health record (EHR) datasets used to train a data-driven predictive model of inpatient admission orders. The second paper presents a decision support tool for surgical team selection, relying on the history of surgical team members and the specific characteristics of the patient. The third paper compares three commercial drug-drug interaction knowledge bases, particularly against a reference list of highly-significant known interactions. The fourth paper focuses on supporting the diagnosis of postoperative delirium using an adaptation of the “anchor and learn” framework, which was applied in unstructured texts contained in EHRs.

Conclusions:  The conducted review illustrated also this year that research in the field of CDSS is very active. Of note is the increase in publications concerning data-driven CDSSs, as revealed by the review process and also reflected by the four papers that have been selected. This trend is in line with the current attention that “Big Data” and data-driven artificial intelligence have gained in the domain of health and CDSSs in particular.

Outpatient CPOE orders discontinued due to 'erroneous entry': prospective survey of prescribers' explanations for errors

BACKGROUND

Computerised prescriber order entry (CPOE) systems users often discontinue medications because the initial order was erroneous.

OBJECTIVE

To elucidate error types by querying prescribers about their reasons for discontinuing outpatient medication orders that they had self-identified as erroneous.

METHODS

During a nearly 3 year retrospective data collection period, we identified 57 972 drugs discontinued with the reason 'Error (erroneous entry)." Because chart reviews revealed limited information about these errors, we prospectively studied consecutive, discontinued erroneous orders by querying prescribers in near-real-time to learn more about the erroneous orders.

RESULTS

From January 2014 to April 2014, we prospectively emailed prescribers about outpatient drug orders that they had discontinued due to erroneous initial order entry. Of 2 50 806 medication orders in these 4 months, 1133 (0.45%) of these were discontinued due to error. From these 1133, we emailed 542 unique prescribers to ask about their reason(s) for discontinuing these mediation orders in error. We received 312 responses (58% response rate). We categorised these responses using a previously published taxonomy. The top reasons for these discontinued erroneous orders included: medication ordered for wrong patient (27.8%, n=60); wrong drug ordered (18.5%, n=40); and duplicate order placed (14.4%, n=31). Other common discontinued erroneous orders related to drug dosage and formulation (eg, extended release versus not). Oxycodone (3%) was the most frequent drug discontinued error.

CONCLUSION

Drugs are not infrequently discontinued 'in error.' Wrong patient and wrong drug errors constitute the leading types of erroneous prescriptions recognised and discontinued by prescribers. Data regarding erroneous medication entries represent an important source of intelligence about how CPOE systems are functioning and malfunctioning, providing important insights regarding areas for designing CPOE more safely in the future.

Reasons for computerised provider order entry (CPOE)-based inpatient medication ordering errors: an observational study of voided orders

OBJECTIVE

Medication voiding is a computerised provider order entry (CPOE)-based discontinuation mechanism that allows clinicians to identify erroneous medication orders. We investigated the accuracy of voiding as an indicator of clinician identification and interception of a medication ordering error, and investigated reasons and root contributors for medication ordering errors.

METHOD

Using voided orders identified with a void alert, we conducted interviews with ordering and voiding clinicians, followed by patient chart reviews. A structured coding framework was used to qualitatively analyse the reasons for medication ordering errors. We also compared clinician-CPOE-selected (at time of voiding), clinician-reported (interview) and chart review-based reasons for voiding.

RESULTS

We conducted follow-up interviews on 101 voided orders. The positive predictive value (PPV) of voided orders that were medication ordering errors was 93.1% (95% CI 88.1% to 98.1%, n=94). Using chart review-based reasons as the gold standard, we found that clinician-CPOE-selected reasons were less reflective (PPV=70.2%, 95% CI 61.0% to 79.4%) than clinician-reported (interview) (PPV=86.1%, 95%CI 78.2% to 94.1%) reasons for medication ordering errors. Duplicate (n=44) and improperly composed (n=41) ordering errors were common, often caused by predefined order sets and data entry issues. A striking finding was the use of intentional violations as a mechanism to notify and seek ordering assistance from pharmacy service. Nearly half of the medication ordering errors were voided by pharmacists.

DISCUSSION

We demonstrated that voided orders effectively captured medication ordering errors. The mismatch between clinician-CPOE-selected and the chart review-based reasons for error emphasises the need for developing standardised operational descriptions for medication ordering errors. Such standardisation can help in accurately identifying, tracking, managing and sharing erroneous orders and their root contributors between healthcare institutions, and with patient safety organisations.