Evaluation of an automated test ordering and feedback system for general practitioners in daily practice

The GUIDES checklist: development of a tool to improve the successful use of guideline-based computerised clinical decision support

Background

Computerised decision support (CDS) based on trustworthy clinical guidelines is a key component of a learning healthcare system. Research shows that the effectiveness of CDS is mixed.

Multifaceted context, system, recommendation and implementation factors may potentially affect the success of CDS interventions. This paper describes the development of a checklist that is intended to support professionals to implement CDS successfully.

Methods

We developed the checklist through an iterative process that involved a systematic review of evidence and frameworks, a synthesis of the success factors identified in the review, feedback from an international expert panel that evaluated the checklist in relation to a list of desirable framework attributes, consultations with patients and healthcare consumers and pilot testing of the checklist.

Results

We screened 5347 papers and selected 71 papers with relevant information on success factors for guideline-based CDS. From the selected papers, we developed a 16-factor checklist that is divided in four domains, i.e. the CDS context, content, system and implementation domains. The panel of experts evaluated the checklist positively as an instrument that could support people implementing guideline-based CDS across a wide range of settings globally. Patients and healthcare consumers identified guideline-based CDS as an important quality improvement intervention and perceived the GUIDES checklist as a suitable and useful strategy.

Conclusions

The GUIDES checklist can support professionals in considering the factors that affect the success of CDS interventions. It may facilitate a deeper and more accurate understanding of the factors shaping CDS effectiveness. Relying on a structured approach may prevent that important factors are missed.

Electronic supplementary material

The online version of this article (10.1186/s13012-018-0772-3) contains supplementary material, which is available to authorized users.

The effect of registry-based performance feedback via short text messages and traditional postal letters on prescribing parenteral steroids by general practitioners--A randomized controlled trial

BACKGROUND

It is conjectured that providing feedback on physicians' prescribing behavior improves quality of drug prescriptions. However, the effectiveness of feedback provision and mode of feedback delivery is not well understood. The objective of this study was to assess and compare the effect of traditional paper letters (TPL) and short text message (STM) feedback on general practitioners' prescribing behavior of parenteral steroids (PSs).

METHODS

In a single-blind randomized controlled trial, 906 general practitioners (GPs) having at least 10 monthly prescriptions were randomly recruited into two interventions and one control study arms with 1:1 allocation, stratified by percentage of prescriptions. The intervention was the provision of 3 feedback messages containing prescribing indices in TPL and STM (in the first two arms) versus the control arm (CG) with an interval of 3 months between these messages. We calculated the PS Defined Daily Dose (DDD) for every GP, every month, and compared between the 3 arms, before and after the interventions. The expected primary outcome was to reduce prescription of parenteral steroids by participants. The study was performed in the Kerman Social Security Organization in Iran.

RESULTS

A total of 906 GPs were selected for the trial, but only 721 of them (TPL=191, STM=228, CG=302) were recruited for the 1st feedback. The mean age of GPs was 44 and 59% of them were male. The prescribed parenteral steroid DDDs at baseline were similar (TPL=121.62, STM=127.49, CG=115.68, P>0.5). At the end of the study, DDDs in the TPL and STM arms were similar (TPL=104.38, STM=101.90, P>0.9) but DDDs in each intervention arm was statistically significantly lower than in CG (CG=156.17, P<0.0001). Being in TPL and STM arms resulted in 36.1 and 41.7 units of decrease in DDD respectively, compared to the control arm (P<0.02 and P<0.005) after the one-year duration of the study.

CONCLUSION

Feedback by TPLs and STMs on prescribing performance effectively reduced prescribing PSs by GPs. STM, being a cheap and fast tool, is potentially powerful and efficient for drug prescription rationalization.

Evaluating the uptake and effects of the computerized decision support system NHGDoc on quality of primary care: protocol for a large-scale cluster randomized controlled trial

BACKGROUND

Computerized decision support systems (CDSSs) are increasingly used to improve quality of care. There is evidence for moderate to large effects from randomized controlled trials (RCTs), but evidence on their effectiveness when implemented at a national level is lacking. In the Netherlands, the Dutch College of General Practitioners (NHG) initiated their successful guideline program already 30 years ago. NHGDoc, a CDSS based on these NHG guidelines, covering multiple disease areas for general practice, was developed in 2006 with the aim to improve quality of primary care. In this paper, a protocol is presented to evaluate the uptake and effects of NHGDoc.

METHODS

A cluster RCT will be conducted among 120 general practices in the Netherlands. Eligible general practices will be randomized to receive either the regular NHGDoc decision support modules (control arm) or the regular modules plus an additional module on heart failure (intervention arm). The heart failure module consists of patient-specific alerts concerning the treatment of patients with heart failure. The effect evaluation will focus on performance indicators (e.g., prescription behavior) as well as on patient outcomes (e.g., hospital admissions) relevant in the domain of heart failure. Additionally, a process evaluation will be conducted to gain insight into the barriers and facilitators that affect the uptake and impact of NHGDoc.

DISCUSSION

Results of this study will provide insight in the uptake and impact of a multiple-domain covering CDSS for primary care implemented by a national guideline organization to improve the quality of primary care. Whereas the trial focuses on a specific domain of care-heart failure-conclusions of this study will shed light on the functioning of CDSSs covering multiple disease areas for primary care, particularly as this study also explores the factors contributing to the system's uptake and effectiveness.

TRIAL REGISTRATION

Clinical trials NCT01773057.

How to translate therapeutic recommendations in clinical practice guidelines into rules for critiquing physician prescriptions? Methods and application to five guidelines

BACKGROUND

Clinical practice guidelines give recommendations about what to do in various medical situations, including therapeutical recommendations for drug prescription. An effective way to computerize these recommendations is to design critiquing decision support systems, i.e. systems that criticize the physician's prescription when it does not conform to the guidelines. These systems are commonly based on a list of "if conditions then criticism" rules. However, writing these rules from the guidelines is not a trivial task. The objective of this article is to propose methods that (1) simplify the implementation of guidelines' therapeutical recommendations in critiquing systems by automatically translating structured therapeutical recommendations into a list of "if conditions then criticize" rules, and (2) can generate an appropriate textual label to explain to the physician why his/her prescription is not recommended.

METHODS

We worked on the therapeutic recommendations in five clinical practice guidelines concerning chronic diseases related to the management of cardiovascular risk. We evaluated the system using a test base of more than 2000 cases.

RESULTS

Algorithms for automatically translating therapeutical recommendations into "if conditions then criticize" rules are presented. Eight generic recommendations are also proposed; they are guideline-independent, and can be used as default behaviour for handling various situations that are usually implicit in the guidelines, such as decreasing the dose of a poorly tolerated drug. Finally, we provide models and methods for generating a human-readable textual critique. The system was successfully evaluated on the test base.

CONCLUSION

We show that it is possible to criticize physicians' prescriptions starting from a structured clinical guideline, and to provide clear explanations. We are now planning a randomized clinical trial to evaluate the impact of the system on practices.

The effects of on-screen, point of care computer reminders on processes and outcomes of care

BACKGROUND

The opportunity to improve care by delivering decision support to clinicians at the point of care represents one of the main incentives for implementing sophisticated clinical information systems. Previous reviews of computer reminder and decision support systems have reported mixed effects, possibly because they did not distinguish point of care computer reminders from e-mail alerts, computer-generated paper reminders, and other modes of delivering 'computer reminders'.

OBJECTIVES

To evaluate the effects on processes and outcomes of care attributable to on-screen computer reminders delivered to clinicians at the point of care.

SEARCH STRATEGY

We searched the Cochrane EPOC Group Trials register, MEDLINE, EMBASE and CINAHL and CENTRAL to July 2008, and scanned bibliographies from key articles.

SELECTION CRITERIA

Studies of a reminder delivered via a computer system routinely used by clinicians, with a randomised or quasi-randomised design and reporting at least one outcome involving a clinical endpoint or adherence to a recommended process of care.

DATA COLLECTION AND ANALYSIS

Two authors independently screened studies for eligibility and abstracted data. For each study, we calculated the median improvement in adherence to target processes of care and also identified the outcome with the largest such improvement. We then calculated the median absolute improvement in process adherence across all studies using both the median outcome from each study and the best outcome.

MAIN RESULTS

Twenty-eight studies (reporting a total of thirty-two comparisons) were included. Computer reminders achieved a median improvement in process adherence of 4.2% (interquartile range (IQR): 0.8% to 18.8%) across all reported process outcomes, 3.3% (IQR: 0.5% to 10.6%) for medication ordering, 3.8% (IQR: 0.5% to 6.6%) for vaccinations, and 3.8% (IQR: 0.4% to 16.3%) for test ordering. In a sensitivity analysis using the best outcome from each study, the median improvement was 5.6% (IQR: 2.0% to 19.2%) across all process measures and 6.2% (IQR: 3.0% to 28.0%) across measures of medication ordering. In the eight comparisons that reported dichotomous clinical endpoints, intervention patients experienced a median absolute improvement of 2.5% (IQR: 1.3% to 4.2%). Blood pressure was the most commonly reported clinical endpoint, with intervention patients experiencing a median reduction in their systolic blood pressure of 1.0 mmHg (IQR: 2.3 mmHg reduction to 2.0 mmHg increase).

AUTHORS' CONCLUSIONS

Point of care computer reminders generally achieve small to modest improvements in provider behaviour. A minority of interventions showed larger effects, but no specific reminder or contextual features were significantly associated with effect magnitude. Further research must identify design features and contextual factors consistently associated with larger improvements in provider behaviour if computer reminders are to succeed on more than a trial and error basis.

A randomized trial of electronic clinical reminders to improve medication laboratory monitoring

OBJECTIVE

Recommendations for routine laboratory monitoring to reduce the risk of adverse medication events are not consistently followed. We evaluated the impact of electronic reminders delivered to primary care physicians on rates of appropriate routine medication laboratory monitoring.

DESIGN

We enrolled 303 primary care physicians caring for 1,922 patients across 20 ambulatory clinics that had at least one overdue routine laboratory test for a given medication between January and June 2004. Clinics were randomized so that physicians received either usual care or electronic reminders at the time of office visits focused on potassium, creatinine, liver function, thyroid function, and therapeutic drug levels.

MEASUREMENTS

Primary outcomes were the receipt of recommended laboratory monitoring within 14 days following an outpatient clinic visit. The effect of the intervention was assessed for each reminder after adjusting for clustering within clinics, as well as patient and provider characteristics.

RESULTS

Medication-laboratory monitoring non-compliance ranged from 1.6% (potassium monitoring with potassium-supplement use) to 6.3% (liver function monitoring with HMG CoA Reductase Inhibitor use). Rates of appropriate laboratory monitoring following an outpatient visit ranged from 14% (therapeutic drug levels) to 64% (potassium monitoring with potassium-sparing diuretic use). Reminders for appropriate laboratory monitoring had no impact on rates of receiving appropriate testing for creatinine, potassium, liver function, renal function, or therapeutic drug level monitoring.

CONCLUSION

We identified high rates of appropriate laboratory monitoring, and electronic reminders did not significantly improve these monitoring rates. Future studies should focus on settings with lower baseline adherence rates and alternate drug-laboratory combinations.

Feasibility and acceptability of a computerised system with automated reminders for prescribing behaviour in primary care

OBJECTIVE

To evaluate the feasibility and acceptability of a computer reminder system (CRS) to improve prescribing behaviour in general practice and to explore the strengths and weaknesses of a reminder system. One group of GPs received reminders on cholesterol lowering drugs, the other group on antibiotics, asthma and COPD drugs.

METHODS

Process evaluation of the computer reminder system being used by 53 GPs in 20 practices, by means of an analysis of the research database of the CRS. In addition, a questionnaire and semi-structured face-to-face interview were conducted with all GP practices, two project leaders, and one technical consultant.

RESULTS

The strategy was largely carried out as planned, although the development period for the CRS had to be extended. Nine percent of the GPs dropped out. We found a significant learning curve without extinguishing effect (p=0.03) for the antibiotics reminders. The questionnaire showed that, in general, GPs were satisfied with the user-friendliness and the content of the different types of reminders, but less satisfied with certain specific technical performance issues of the system. The GPs reported mixed feelings towards the CRS in the interviews. They were generally positive about the guidelines themselves, but negative regarding to the organisational context and the method of implementing the CRS. GPs stated that they sometimes manipulated the system to bypass reminders. Interviews with the project leaders and technical consultant revealed barriers to cooperation and miscommunication between the different parties, and technical problems with multiple updates of the GP information system and the operating system.

CONCLUSIONS

This process evaluation demonstrated that the implementation of the CRS was mainly carried out as planned, but the subjective experience of working with the CRS was not only positive. Participating GPs had mixed feelings, and quite a number of barriers need to be addressed to facilitate large-scale implementation of the CRS. Costs cannot be neglected, so it is important to analyse the balance between costs and effects.

Cost consequences of implementing an electronic decision support system for ordering laboratory tests in primary care: evidence from a controlled prospective study in the Netherlands

BACKGROUND

The economic consequences of interventions to promote rational, evidence-based use of laboratory tests by physicians are not yet fully understood. We evaluated the cost consequences of a computer-based, guideline-driven decision-support system (CDSS) for ordering blood tests in primary care.

METHODS

We installed the CDSS in 118 practices [159 general practitioners (GPs)] throughout The Netherlands and calculated the costs of the intervention in this group. During a period of 6 months before and 6 months after installation of the CDSS, the test-ordering behavior of 87 (109 GPs) of these 118 study practices was studied and the results were compared with those of a nonhistorical control group that did not receive the CDSS. In addition the costs of laboratory requests were calculated for both groups.

RESULTS

Total intervention costs, comprising development costs and installation costs, amounted to 79,000 euro (670 euro per practice). Whereas the introduction of the CDSS did not affect the number of order forms submitted to the laboratories, it did reduce the number of blood tests per order form. As a result, the CDSS yielded mean savings on the costs of laboratory requests of 847 euro per practice per 6 months.

CONCLUSIONS

This study demonstrates that providing electronic decision support for ordering blood tests in primary care represents an economically promising concept. Savings on laboratory costs are achievable and not offset by disproportionally high intervention costs.