Evaluation of features to support safety and quality in general practice clinical software

The Effect of Cognitive Load and Task Complexity on Automation Bias in Electronic Prescribing

OBJECTIVE

Determine the relationship between cognitive load (CL) and automation bias (AB).

BACKGROUND

Clinical decision support (CDS) for electronic prescribing can improve safety but introduces the risk of AB, where reliance on CDS replaces vigilance in information seeking and processing. We hypothesized high CL generated by high task complexity would increase AB errors.

METHOD

One hundred twenty medical students prescribed medicines for clinical scenarios using a simulated e-prescribing system in a randomized controlled experiment. Quality of CDS (correct, incorrect, and no CDS) and task complexity (low and high) were varied. CL, omission errors (failure to detect prescribing errors), and commission errors (acceptance of false positive alerts) were measured.

RESULTS

Increasing complexity from low to high significantly increased CL, F(1, 118) = 71.6, p < .001. CDS reduced CL in high-complexity conditions compared to no CDS, F(2, 117) = 4.72, p = .015. Participants who made omission errors in incorrect and no CDS conditions exhibited lower CL compared to those who did not, F(1, 636.49) = 3.79, p = .023.

CONCLUSION

Results challenge the notion that AB is triggered by increasing task complexity and associated increases in CL. Omission errors were associated with lower CL, suggesting errors may stem from an insufficient allocation of cognitive resources.

APPLICATION

This is the first research to examine the relationship between CL and AB. Findings suggest designers and users of CDS systems need to be aware of the risks of AB. Interventions that increase user vigilance and engagement may be beneficial and deserve further investigation.

Supporting general practice to make timely decisions for better health care: a population health approach

Primary Health Networks (PHNs) are tasked to enhance the efficiency and effectiveness of general practice. Gold Coast Primary Health Network (GCPHN) has been collecting de-identified aggregated clinical data from general practices and reporting back on areas for improvement on data coding and some clinical metrics, such as blood pressure not being recorded. However, aggregated data cannot be used to intervene at the individual patient level, and because of the collection-to-reporting time-lag, the data cannot help facilitate immediate action in the general practice. GCPHN developed a practice-based population health management approach based on mapping data from general practices to international coding classification systems, and applying evidenced-based algorithms and tools. General practices are provided with a health profile of their entire patient population, from the healthiest to the most complex comorbid patients. The information is conveyed as alerts and reports on areas including medication quality and safety, possible gaps in care and high-risk patients. The information is received based on the practice’s preferences; this can be at the time of data entry, the following day or as specified. Strong clinical governance has ensured GCPHN’s approach and methodologies are evidenced-based and appropriate. The consistent application of clinical governance within general practices is also needed to ensure the approach is sustainable and improves clinical outcomes.

Automation bias in electronic prescribing

Background

Clinical decision support (CDS) in e-prescribing can improve safety by alerting potential errors, but introduces new sources of risk. Automation bias (AB) occurs when users over-rely on CDS, reducing vigilance in information seeking and processing. Evidence of AB has been found in other clinical tasks, but has not yet been tested with e-prescribing. This study tests for the presence of AB in e-prescribing and the impact of task complexity and interruptions on AB.

Methods

One hundred and twenty students in the final two years of a medical degree prescribed medicines for nine clinical scenarios using a simulated e-prescribing system. Quality of CDS (correct, incorrect and no CDS) and task complexity (low, low + interruption and high) were varied between conditions. Omission errors (failure to detect prescribing errors) and commission errors (acceptance of false positive alerts) were measured.

Results

Compared to scenarios with no CDS, correct CDS reduced omission errors by 38.3% (p < .0001, n = 120), 46.6% (p < .0001, n = 70), and 39.2% (p < .0001, n = 120) for low, low + interrupt and high complexity scenarios respectively. Incorrect CDS increased omission errors by 33.3% (p < .0001, n = 120), 24.5% (p < .009, n = 82), and 26.7% (p < .0001, n = 120). Participants made commission errors, 65.8% (p < .0001, n = 120), 53.5% (p < .0001, n = 82), and 51.7% (p < .0001, n = 120). Task complexity and interruptions had no impact on AB.

Conclusions

This study found evidence of AB omission and commission errors in e-prescribing. Verification of CDS alerts is key to avoiding AB errors. However, interventions focused on this have had limited success to date. Clinicians should remain vigilant to the risks of CDS failures and verify CDS.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-017-0425-5) contains supplementary material, which is available to authorized users.

Validation of a Crowdsourcing Methodology for Developing a Knowledge Base of Related Problem-Medication Pairs

BACKGROUND

Clinical knowledge bases of problem-medication pairs are necessary for many informatics solutions that improve patient safety, such as clinical summarization. However, developing these knowledge bases can be challenging.

OBJECTIVE

We sought to validate a previously developed crowdsourcing approach for generating a knowledge base of problem-medication pairs in a large, non-university health care system with a widely used, commercially available electronic health record.

METHODS

We first retrieved medications and problems entered in the electronic health record by clinicians during routine care during a six month study period. Following the previously published approach, we calculated the link frequency and link ratio for each pair then identified a threshold cutoff for estimated problem-medication pair appropriateness through clinician review; problem-medication pairs meeting the threshold were included in the resulting knowledge base. We selected 50 medications and their gold standard indications to compare the resulting knowledge base to the pilot knowledge base developed previously and determine its recall and precision.

RESULTS

The resulting knowledge base contained 26,912 pairs, had a recall of 62.3% and a precision of 87.5%, and outperformed the pilot knowledge base containing 11,167 pairs from the previous study, which had a recall of 46.9% and a precision of 83.3%.

CONCLUSIONS

We validated the crowdsourcing approach for generating a knowledge base of problem-medication pairs in a large non-university health care system with a widely used, commercially available electronic health record, indicating that the approach may be generalizable across healthcare settings and clinical systems. Further research is necessary to better evaluate the knowledge, to compare crowdsourcing with other approaches, and to evaluate if incorporating the knowledge into electronic health records improves patient outcomes.

Safety of medication use in primary care

BACKGROUND

Medication errors are one of the leading causes of harmin health care. Review and analysis of errors have often emphasized their preventable nature and potential for reoccurrence. Of the few error studies conducted in primary care to date, most have focused on evaluating individual parts of the medicines management system. Studying individual parts of the system does not provide a complete perspective and may further weaken the evidence and undermine interventions.

AIM AND OBJECTIVES

The aim of this review is to estimate the scale of medication errors as a problem across the medicines management system in primary care. Objectives were: To review studies addressing the rates of medication errors, and To identify studies on interventions to prevent medication errors in primary care.

METHODS

A systematic search of the literature was performed in PubMed (MEDLINE), International Pharmaceutical Abstracts (IPA), Embase, PsycINFO, PASCAL, Science Direct, Scopus, Web of Knowledge, and CINAHL PLUS from 1999 to November, 2012. Bibliographies of relevant publications were searched for additional studies.

KEY FINDINGS

Thirty-three studies estimating the incidence of medication errors and thirty-six studies evaluating the impact of error-prevention interventions in primary care were reviewed. This review demonstrated that medication errors are common, with error rates between <1% and >90%, depending on the part of the system studied, and the definitions and methods used. The prescribing stage is the most susceptible, and that the elderly (over 65 years), and children (under 18 years) are more likely to experience significant errors. Individual interventions demonstrated marginal improvements in medication safety when implemented on their own.

CONCLUSION

Targeting the more susceptible population groups and the most dangerous aspects of the system may be a more effective approach to error management and prevention. Co-implementation of existing interventions at points within the system may offer time- and cost-effective options to improving medication safety in primary care.

Adherence factors to paperless primary healthcare: a focus group study

Background: Information and communication technology (ICT) and paperless practices have been shown to improve “existing processes in the workplace” “as well as being an important component of modern primary healthcare”. The aim of our study was to analyse the attitudes of health-care professionals and patients with regard to paperless practice and the most frequently used information and communication technology tools in Slovenian primary healthcare.

Methods and participants: Qualitative methodology using focus groups of 22 primary care physicians, 14 nurses and 18 patients.

Results: The areas recognised by all participants as important for further information and communication technology development were: computer-supported decision making, accessibility and completeness of personal e-health data, emergency cases, support for chronic disease management, ICT related time savings, e-prescriptions and e-discharge letters. The most important identified barriers impeding the use of ICT were: the heavy workload of primary care physicians and nurses, health insurance reimbursement rules and duplication of work using both paper and electronic health records.

Conclusions: This study highlighted a number of strengths of ICT use in primary care as well as numerous areas where changes in procedures and improvement of ICT tools to support them are needed.

Differentiating innovation priorities among stakeholder in hospital care

BACKGROUND

Decisions to adopt a particular innovation may vary between stakeholders because individual stakeholders may disagree on the costs and benefits involved. This may translate to disagreement between stakeholders on priorities in the implementation process, possibly explaining the slow diffusion of innovations in health care. In this study, we explore the differences in stakeholder preferences for innovations, and quantify the difference in stakeholder priorities regarding costs and benefits.

METHODS

The decision support technique called the analytic hierarchy process was used to quantify the preferences of stakeholders for nine information technology (IT) innovations in hospital care. The selection of the innovations was based on a literature review and expert judgments. Decision criteria related to the costs and benefits of the innovations were defined. These criteria were improvement in efficiency, health gains, satisfaction with care process, and investments required. Stakeholders judged the importance of the decision criteria and subsequently prioritized the selected IT innovations according to their expectations of how well the innovations would perform for these decision criteria.

RESULTS

The stakeholder groups (patients, nurses, physicians, managers, health care insurers, and policy makers) had different preference structures for the innovations selected. For instance, self-tests were one of the innovations most preferred by health care insurers and managers, owing to their expected positive impacts on efficiency and health gains. However, physicians, nurses and patients strongly doubted the health gains of self-tests, and accordingly ranked self-tests as the least-preferred innovation.

CONCLUSIONS

The various stakeholder groups had different expectations of the value of the nine IT innovations. The differences are likely due to perceived stakeholder benefits of each innovation, and less to the costs to individual stakeholder groups. This study provides a first exploratory quantitative insight into stakeholder positions concerning innovation in health care, and presents a novel way to study differences in stakeholder preferences. The results may be taken into account by decision makers involved in the implementation of innovations.

Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications

OBJECTIVE

We describe a novel, crowdsourcing method for generating a knowledge base of problem-medication pairs that takes advantage of manually asserted links between medications and problems.

METHODS

Through iterative review, we developed metrics to estimate the appropriateness of manually entered problem-medication links for inclusion in a knowledge base that can be used to infer previously unasserted links between problems and medications.

RESULTS

Clinicians manually linked 231,223 medications (55.30% of prescribed medications) to problems within the electronic health record, generating 41,203 distinct problem-medication pairs, although not all were accurate. We developed methods to evaluate the accuracy of the pairs, and after limiting the pairs to those meeting an estimated 95% appropriateness threshold, 11,166 pairs remained. The pairs in the knowledge base accounted for 183,127 total links asserted (76.47% of all links). Retrospective application of the knowledge base linked 68,316 medications not previously linked by a clinician to an indicated problem (36.53% of unlinked medications). Expert review of the combined knowledge base, including inferred and manually linked problem-medication pairs, found a sensitivity of 65.8% and a specificity of 97.9%.

CONCLUSION

Crowdsourcing is an effective, inexpensive method for generating a knowledge base of problem-medication pairs that is automatically mapped to local terminologies, up-to-date, and reflective of local prescribing practices and trends.