Coded entry versus free-text and alert overrides: what you get depends on how you ask

Comprehensibility of Contraindications in German, UK and US Summaries of Product Characteristics/Prescribing Information—A Comparative Qualitative and Quantitative Analysis

Contraindications (CIs) in Summaries of Product Characteristics (SmPCs)/Prescribing Information (PI) that lack clarity may pose a risk to medication safety and increase the risk for adverse drug reactions. We assessed and compared SmPCs/PI from three major drug markets regarding comprehensibility from the prescriber perspective, as well as usability in clinical decision support systems. 158 drugs met the following inclusion criteria: marketed in Germany (DE), United Kingdom (UK) and United States (US) and belonged to the 100 most recently FDA approved and/or 100 most frequently prescribed drugs in either country. In the 474 (3 × 158) SmPCs/PI all expressions for absolute CIs were identified, divided into 3999 stand-alone terms and evaluated according to ‘clarity’ and ‘codability’. The average number of absolute CIs per drug differed drastically between the three markets (DE: 11.7, UK: 9.0, US: 4.6). Expressions were frequently unclear (DE: 27.2% (95% CI 25.2–29.2%), UK: 28.5% (26.2–30.9%), US: 22.6% (19.7–25.8%)). Moreover, 60.9% (58.6–63.1%), 63.6% (61.0–66.0%), and 64.7% (61.2–68.1%) of the expressions were not codable in DE, UK, and US, respectively. Taken together, in three major drug markets, statements regarding CIs in SmPCs/PI substantially differ in frequency and frequently lack clarity and codability which poses an unnecessary obstacle to medication safety.

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons

Background

Rule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited.

Objective

Investigate whether user override comments can be used to discover malfunctions.

Methods

We manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: “broken,” “not broken, but could be improved,” and “not broken.” We used 3 methods (frequency of comments, cranky word list heuristic, and a Naïve Bayes classifier trained on a sample of comments) to automatically rank rules based on features of their override comments. We evaluated each ranking using the manual classification as truth.

Results

Of the rules investigated, 62 were broken, 13 could be improved, and the remaining 45 were not broken. Frequency of comments performed worse than a random ranking, with precision at 20 of 8 and AUC = 0.487. The cranky comments heuristic performed better with precision at 20 of 16 and AUC = 0.723. The Naïve Bayes classifier had precision at 20 of 17 and AUC = 0.738.

Discussion

Override comments uncovered malfunctions in 26% of all rules active in our system. This is a lower bound on total malfunctions and much higher than expected. Even for low-resource organizations, reviewing comments identified by the cranky word list heuristic may be an effective and feasible way of finding broken alerts.

Conclusion

Override comments are a rich data source for finding alerts that are broken or could be improved. If possible, we recommend monitoring all override comments on a regular basis.

Categorization of free-text drug orders using character-level recurrent neural networks

BACKGROUND AND PURPOSE

Manual annotation and categorization of non-standardized text ("free-text") of drug orders entered into electronic health records is a labor-intensive task. However, standardization is required for drug order analyses and has implications for clinical decision support. Machine learning could help to speed up manual labelling efforts. The objective of this study was to analyze the performance of deep machine learning methods to annotate non-standardized text of drug order entries with their therapeutically active ingredients.

MATERIALS AND METHODS

The data consisted of drug orders entered 8/2009-4/2014 into the electronic health records of inpatients at a large tertiary care academic medical center. We manually annotated the most frequent order entry patterns with the active ingredient they contain (e.g. "Prograf"⟵"Tacrolimus"). We heuristically included additional orders by means of character sequence comparisons to augment the training dataset. Finally, we trained and employed character-level recurrent deep neural networks to classify non-standardized text of drug order entries according to their active ingredients.

RESULTS

A total of 26,611 distinct order patterns were considered in our study, of which the top 7.6% (2028) had been annotated with one of 558 distinct ingredients, leaving 24,583 unlabeled observations. Character-level recurrent deep neural networks achieved a Mean Reciprocal Rank (MRR) of 98% and outperformed the best representative baseline, a trigram-based Support Vector Machine, by 2 percentage points.

CONCLUSION

Character-level recurrent deep neural networks can be used to map the active ingredient to non-standardized text of drug order entries, outperforming other representative techniques. While machine learning might help to facilitate categorization tasks, still a considerable amount of manual labelling and reviewing work is required to train such systems.

Influence of pharmacist intervention on drug safety of geriatric inpatients: a prospective, controlled trial

BACKGROUND

Demographic shift leads to an increasing number of geriatric patients suffering from multimorbidity and resulting polypharmacy. Polypharmacy is shown to be associated with drug-related problems (DRPs) and increased morbidity. For Germany, a hospital-based intervention may be successful optimizing of polypharmacy. The aim of this study was to reduce DRPs in geriatric inpatients by a structured pharmacist's intervention and to measure the acceptance rate of pharmaceutical recommendations.

METHODS

This study followed an open, prospective, quasi-randomized, controlled design and was conducted in a geriatric department in a teaching hospital in Germany. Patients of all sexes were included, with a minimum age of 70 years, a written informed consent and a regular intake of at least five drugs daily. Primary outcome was the percentage of patients having a DRP at admission and discharge. A DRP was defined as a prescription without indication or a relevant drug-drug interaction or prescription of a potentially inappropriate medication or presence of an adverse drug reaction. Recommendations were classified and discussed face to face. Statistical analyses were performed using a full-set analysis and a matched-pairs design.

RESULTS

Within 12 months, 411 patients were recruited with median age of 82 years (intervention: n = 209; control: n = 202). Median number of drugs at admission was 10 (range 5-24), at discharge 9 (range 3-21). In the intervention group, the percentage of patients with a DRP was reduced from 86.6% to 56.0%; in the control group, from 76.7% to 76.2% (p value < 0.001). Medication appropriateness index score was reduced by 56% in the intervention group and by 0.2% in the control group (p value < 0.001). Implementation rate of the pharmaceutical recommendation was 80%.

CONCLUSION

This prospective controlled trial showed that a pharmacist's intervention was successful in optimizing polypharmacy in geriatric inpatients.

The lexeme hypotheses: Their use to generate highly grammatical and completely computerized medical records

INTRODUCTION

Medical records often contain free text created by harried clinicians. Free text often contains errors which make it an unsuitable target for computerized data extraction. The cost of healthcare can be reduced by creating medical records that are fully computerized at their inception. We examine hypotheses that enable us to construct such records.

METHODS

We regard the text of the medical record as being an ordered collection of meaningful fragments. The intellectual content (or "lexeme") of each text fragment in the record is considered separately from the language that used to express it. We further consider that each lexeme exists as a combination of a lexeme query (defining the issue being addressed) and a lexeme response to that query. The medical record can then be perceived as a stream of these responses. The responses can be expressed in any style or language, including computer code. Examining medical records in this light gives rise to a number of observations and hypotheses.

OBSERVATIONS AND HYPOTHESES

The physical location and nature of the medical episode (which we term "context") determines the general layout of the record. The order that lexeme-queries are addressed in within the record is highly consistent ("coherence"). Issues are only addressed if they are logically called-for by the context or by a previously-selected lexeme response ("predicance"), and only to a needed depth of detail ("level"). We hypothesize that all of the lexeme queries required to write any clinical notes can be stored in a large database ("lexicon") in coherence order, wherein each lexeme query is associated with its own collection of lexeme responses. We hypothesize that the issue a note-writer will need to address next is identifiable purely by using the rules of coherence, level and predicance.

TESTING THE HYPOTHESES AND THEIR UTILITY

We have tested these hypotheses with a computer program which repeatedly offers the user a menu of lexeme responses with associated text. On selection, the program issues the text fragment, and its corresponding computer code, to output files. The program then uses coherence, predicance and level to navigate to the next appropriate lexeme query for presentation to the user. The net result is that the user creates a grammatically correct and completely computerized note at the time of its inception. The value of this approach and its practical implementation to create medical records are discussed. In our work so far, the hypotheses appear not to be false, but further testing is needed using a larger lexicon to establish their robustness in actual clinical practice.

Increased appropriateness of customized alert acknowledgement reasons for overridden medication alerts in a computerized provider order entry system

OBJECTIVE

Computerized provider order entry systems commonly contain alerting mechanisms for patient allergies, incorrect doses, or drug-drug interactions when ordering medications. Providers have the option to override (bypass) these alerts and continue with the order unchanged. This study examines the effect of customizing medication alert override options on the appropriateness of override selection related to patient allergies, drug dosing, and drug-drug interactions when ordering medications in an electronic medical record.

MATERIALS AND METHODS

In this prospective, randomized crossover study, providers were randomized into cohorts that required a reason for overriding a medication alert from a customized or non-customized list of override reasons and/or by free-text entry. The primary outcome was to compare override responses that appropriately correlate with the alert type between the customized and non-customized configurations. The appropriateness of a subset of free-text responses that represented an affirmative and active acknowledgement of the alert without further explanation was classified as "indeterminate." Results were analyzed in three different ways by classifying indeterminate answers as either appropriate, inappropriate, or excluded entirely. Secondary outcomes included the appropriateness of override reasons when comparing cohorts and individual providers, reason selection based on order within the override list, and the determination of the frequency of free-text use, nonsensical responses, and multiple selection responses.

RESULTS

Twenty-two clinicians were randomized into 2 cohorts and a total of 1829 alerts with a required response were generated during the study period. The customized configuration had a higher rate of appropriateness when compared to the non-customized configuration regardless of how indeterminate responses were classified (p<0.001). When comparing cohorts, appropriateness was significantly higher in the customized configuration regardless of the classification of indeterminate responses (p<0.001) with one exception: when indeterminate responses were considered inappropriate for the cohort of providers that were first exposed to the non-customized list (p=0.103). Free-text use was higher in the customized configuration overall (p<0.001), and there was no difference in nonsensical response between configurations (p=0.39).

CONCLUSION

There is a benefit realized by using a customized list for medication override reasons. Poor application design or configuration can negatively affect provider behavior when responding to important medication alerts.

Tailoring of alerts substantially reduces the alert burden in computerized clinical decision support for drugs that should be avoided in patients with renal disease

OBJECTIVE

Electronic alerts are often ignored by physicians, which is partly due to the large number of unspecific alerts generated by decision support systems. The aim of the present study was to analyze critical drug prescriptions in a university-based nephrology clinic and to evaluate the effect of different alerting strategies on the alert burden.

METHODS

In a prospective observational study, two advanced strategies to automatically generate alerts were applied when medication regimens were entered for discharge letters, outpatient clinic letters, and written prescriptions and compared to two basic reference strategies. Strategy A generated alerts whenever drug-specific information was available, whereas strategy B generated alerts only when the estimated glomerular filtration rate of a patient was below a drug-specific value. Strategies C and D included further patient characteristics and drug-specific information to generate even more specific alerts.

RESULTS

Overall, 1012 medication regimens were entered during the observation period. The average number of alerts per drug preparation in medication regimens entered for letters was 0.28, 0.080, 0.019, and 0.011, when using strategy A, B, C, or D (P<0.001, for comparison between the strategies), leading to at least one alert in 87.5%, 39.3%, 13.5%, or 7.81 % of the regimens. Similar average numbers of alerts were observed for medication regimens entered for written prescriptions.

CONCLUSIONS

The prescription of potentially hazardous drugs is common in patients with renal impairment. Alerting strategies including patient and drug-specific information to generate more specific alerts have the potential to reduce the alert burden by more than 90 %.

Drug interaction alert override rates in the Meaningful Use era: no evidence of progress

BACKGROUND

Interruptive drug interaction alerts may reduce adverse drug events and are required for Stage I Meaningful Use attestation. For the last decade override rates have been very high. Despite their widespread use in commercial EHR systems, previously described interventions to improve alert frequency and acceptance have not been well studied.

OBJECTIVES

(1) To measure override rates of inpatient medication alerts within a commercial clinical decision support system, and assess the impact of local customization efforts. (2) To compare override rates between drug-drug interaction and drug-allergy interaction alerts, between attending and resident physicians, and between public and academic hospitals. (3) To measure the correlation between physicians' individual alert quantities and override rates as an indicator of potential alert fatigue.

METHODS

We retrospectively analyzed physician responses to drug-drug and drug-allergy interaction alerts, as generated by a common decision support product in a large teaching hospital system.

RESULTS

(1) Over four days, 461 different physicians entered 18,354 medication orders, resulting in 2,455 visible alerts; 2,280 alerts (93%) were overridden. (2) The drug-drug alert override rate was 95.1%, statistically higher than the rate for drug-allergy alerts (90.9%) (p < 0.001). There was no significant difference in override rates between attendings and residents, or between hospitals. (3) Physicians saw a mean of 1.3 alerts per day, and the number of alerts per physician was not significantly correlated with override rate (R2 = 0.03, p = 0.41).

CONCLUSIONS

Despite intensive efforts to improve a commercial drug interaction alert system and to reduce alerting, override rates remain as high as reported over a decade ago. Alert fatigue does not seem to contribute. The results suggest the need to fundamentally question the premises of drug interaction alert systems.

What, if all alerts were specific - estimating the potential impact on drug interaction alert burden

PURPOSE

Clinical decision support systems (CDSS) may potentially improve prescribing quality, but are subject to poor user acceptance. Reasons for alert overriding have been identified and counterstrategies have been suggested; however, poor alert specificity, a prominent reason of alert overriding, has not been well addressed. This paper aims at structuring modulators that determine alert specificity and estimating their quantitative impact on alert burden.

METHODS

We developed and summarized optimizing strategies to guarantee the specificity of alerts and applied them to a set of 100 critical and frequent drug interaction (DDI) alerts. Hence, DDI alerts were classified as dynamic, i.e. potentially sensitive to prescription-, co-medication-, or patient-related factors that would change alert severity or render the alert inappropriate compared to static, i.e. always applicable alerts not modulated by cofactors.

RESULTS

Within the subset of 100 critical DDI alerts, only 10 alerts were considered as static and for 7 alerts, relevant factors are not generally available in today's patient charts or their consideration would not impact alert severity. The vast majority, i.e. 83 alerts, might require a decrease in alert severity due to factors related to the prescription (N=13), the co-medication (N=11), individual patient data (N=36), or combinations of them (N=23). Patient-related factors consisted mainly of three lab values, i.e. renal function, potassium, and therapeutic drug monitoring results.

CONCLUSION

This paper outlines how promising the refinement of knowledge bases is in order to increase specificity and decrease alert burden and suggests how to structure knowledge bases to refine DDI alerting.

Dose omissions in hospitalized patients in a UK hospital: an analysis of the relative contribution of adverse drug reactions

BACKGROUND

The omission of charted (prescribed) doses for hospitalized patients is an important problem in the UK. Inappropriate drug omission can clearly lead to harm from lack of therapeutic effect. However, healthcare professionals administering medicines may decide that omission of a dose is appropriate in certain circumstances, e.g. when patients show signs of a possible adverse drug reaction (ADR).

OBJECTIVE

The aim of this study was to characterize dose omissions to understand the factors that influence non-administration of therapy and to determine the proportion of doses that are appropriately omitted due to ADRs.

METHODS

We used data from a bespoke hospital-wide electronic prescribing and administration system at University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK. We extracted data on 6.01 million drug administrations during 2010 and then randomly selected four 7-day periods, concentrating on doses that were charted but not given. Omitted medicines were counted if either there was a charted 'non-administration' (i.e. an active acknowledgement of the omitted dose) or there was no charting of that dose (i.e. no record of either administration or omission). Paused medicines were not counted. When a dose was omitted, staff indicated the reasons for non-administration using codes ('hard coded') or free text in the electronic system. We used both to compare the contribution of different factors, including ADRs, to the total rates of dose omissions.

RESULTS

In the four 7-day periods analysed, 60 763 (12.4%) of the 491 894 charted doses were omitted. The most common code was 'patient refused drug' (45.4%). Only 1.6% of doses were omitted for reasons of patient safety, of which 4 in 1000 omissions were coded as directly due to an ADR.

CONCLUSIONS

Measures to improve the quality of care should seek to reduce dose omissions, but in some cases omission may be rational. Electronic medication administration records allow for detailed analysis of decisions made by healthcare professionals at the point of administration. While dose omissions related to ADRs are uncommon, they are important both for patient safety and for therapeutic decision making.

Data re-entry overload: time for a paradigm shift in maternity IT?

This paper provides an overview of maternity information technology (IT) in Britain, questioning the usability, effectiveness and cost efficiency of the current models of implementation of electronic maternity records. UK experience of hand-held paper obstetric notes and computerized records reveals fundamental problems in the relationship between the two complementary methods of recording maternity data. The assumption that paper records would inevitably be replaced by electronic substitutes has proven false; the rigidity of analysable electronic records has led to immense incompatibility problems. The flexibility of paper records has distinct advantages that have so far not been sufficiently acknowledged. It is suggested that continuing work is needed to encourage the standardization of electronic maternity records, via a new co-creative, co-development approach and continuing international electronic community debate.