Design and pilot evaluation of a system to develop computer-based site-specific practice guidelines from decision models

Building a web-based tool to support clinical decisions in the control of Chlamydia trachomatis and Neisseria gonorrhoeae infections

Chlamydia trachomatis (CT) and Neisseria gonorrhoeae (GC) are the agents of two common, sexually transmitted diseases afflicting women in the United States (http://www.cdc.gov). We designed a novel web-based application that offers simple recommendations to help optimize medical outcomes with CT and GC prevention and control programs. This application takes population groups, prevalence rates, parameters for available screening assays and treatment regimens (costs, sensitivity, and specificity), as well as budget limits as inputs. Its output suggests optimal screening and treatment strategies for selected at-risk groups, commensurate with the clinic's budget allocation. Development of this tool illustrates how a clinical informatics application based on rigorous mathematics might have a significant impact on real-world clinical issues.

Appraisal tools for clinical practice guidelines: a systematic review

INTRODUCTION

Clinical practice guidelines can improve healthcare processes and patient outcomes, but are often of low quality. Guideline appraisal tools aim to help potential guideline users in assessing guideline quality. We conducted a systematic review of publications describing guideline appraisal tools in order to identify and compare existing tools.

METHODS

Among others we searched MEDLINE, EMBASE and the Cochrane Database of Systematic Reviews from 1995 to May 2011 for relevant primary and secondary publications. We also handsearched the reference lists of relevant publications. On the basis of the available literature we firstly generated 34 items to be used in the comparison of appraisal tools and grouped them into thirteen quality dimensions. We then extracted formal characteristics as well as questions and statements of the appraisal tools and assigned them to the items.

RESULTS

We identified 40 different appraisal tools. They covered between three and thirteen of the thirteen possible quality dimensions and between three and 29 of the possible 34 items. The main focus of the appraisal tools were the quality dimensions "evaluation of evidence" (mentioned in 35 tools; 88%), "presentation of guideline content" (34 tools; 85%), "transferability" (33 tools; 83%), "independence" (32 tools; 80%), "scope" (30 tools; 75%), and "information retrieval" (29 tools; 73%). The quality dimensions "consideration of different perspectives" and "dissemination, implementation and evaluation of the guideline" were covered by only twenty (50%) and eighteen tools (45%) respectively.

CONCLUSIONS

Most guideline appraisal tools assess whether the literature search and the evaluation, synthesis and presentation of the evidence in guidelines follow the principles of evidence-based medicine. Although conflicts of interest and norms and values of guideline developers, as well as patient involvement, affect the trustworthiness of guidelines, they are currently insufficiently considered. Greater focus should be placed on these issues in the further development of guideline appraisal tools.

Computer-interpretable clinical guidelines: a methodological review

Clinical practice guidelines (CPGs) aim to improve the quality of care, reduce unjustified practice variations and reduce healthcare costs. In order for them to be effective, clinical guidelines need to be integrated with the care flow and provide patient-specific advice when and where needed. Hence, their formalization as computer-interpretable guidelines (CIGs) makes it possible to develop CIG-based decision-support systems (DSSs), which have a better chance of impacting clinician behavior than narrative guidelines. This paper reviews the literature on CIG-related methodologies since the inception of CIGs, while focusing and drawing themes for classifying CIG research from CIG-related publications in the Journal of Biomedical Informatics (JBI). The themes span the entire life-cycle of CIG development and include: knowledge acquisition and specification for improved CIG design, including (1) CIG modeling languages and (2) CIG acquisition and specification methodologies, (3) integration of CIGs with electronic health records (EHRs) and organizational workflow, (4) CIG validation and verification, (5) CIG execution engines and supportive tools, (6) exception handling in CIGs, (7) CIG maintenance, including analyzing clinician's compliance to CIG recommendations and CIG versioning and evolution, and finally (8) CIG sharing. I examine the temporal trends in CIG-related research and discuss additional themes that were not identified in JBI papers, including existing themes such as overcoming implementation barriers, modeling clinical goals, and temporal expressions, as well as futuristic themes, such as patient-centric CIGs and distributed CIGs.

A comparison between clinical decisions made about lung cancer patients and those inherent in the corresponding Scottish Intercollegiate Guidelines Network (SIGN) guideline

Treatment and survival for patients with lung cancer vary between and within countries. We have undertaken a multifaceted study of a clinical dataset of 635 patients, to see if clinician treatment decisions were being made consistently and in accordance with the appropriate Scottish Intercollegiate Guidelines Network (SIGN) document. Subsequently, we created a dataset of 117 patients who should have undergone surgery according to the SIGN guideline. As analyses of this dataset did not provide clear distinctions between the main treatment groups, a clinician reviewed the case notes and dataset, checking for inconsistencies. The revised dataset was processed by a decision tree algorithm which suggests clinically plausible decisions. Further, statistical analyses compared the 54 patients offered surgery with the 52 who were not. These analyses suggest that there are significant differences: the most discriminating feature is significant co-morbidity (p < 0.001). The article concludes with suggestions for how future guidelines might be enhanced.

An algebraic approach to detect logical inconsistencies in medical appropriateness criteria

In this paper, we present a computerized approach to detect inconsistencies in medical knowledge bases. The method has been applied to a set of medical appropriateness criteria developed for the review of coronary artery disease management. One of the main problems associated to medical appropriateness criteria is to detect logical inconsistencies in the criteria set, a process often manually carried out by health services specialists. In our approach, appropriateness criteria are automatically translated to rules containing propositional variables, using three-valued Łukasiewicz's logic augmented with modal operators to manage uncertainty. The method assigns a polynomial to each of the rules, integrity constraints, and facts from the rule-based set. This rule set is then checked for inconsistencies. The problem of determining if a formula is a tautological consequence of a set of formulae is reduced by our method into an ideal membership problem in computer algebra. Finally, the set of medical appropriateness criteria is represented in a flowchart format that can be disseminated and remotely accessed over Internet, and can be prospectively used for patient care and management. The method reported in this paper can be applied to other knowledge bases represented by means of IF-THEN rules.

Individualizing generic decision models using assessments as evidence

Complex decision models in expert systems often depend upon a number of utilities and subjective probabilities for an individual. Although these values can be estimated for entire populations or demographic subgroups, a model should be customized to the individual's specific parameter values. This process can be onerous and inefficient for practical decisions. We propose an interactive approach for incrementally improving our knowledge about a specific individual's parameter values, including utilities and probabilities, given a decision model and a prior joint probability distribution over the parameter values. We define the concept of value of elicitation and use it to determine dynamically the next most informative elicitation for a given individual. We evaluated the approach using an example model and demonstrate that we can improve the decision quality by focusing on those parameter values most material to the decision.

A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit

OBJECTIVE

To identify a critical appraisal tool for clinical practice guidelines that could serve as a basis for the development of an appraisal tool for clinical pathways.

DESIGN

Systematic review of the literature and personal contacts. Databases searched were: Medline, Embase, and Cinahl. Search terms were: practice guidelines, appraisal, and evaluation. The items of the identified appraisal tools were examined and thematically grouped into 10 guideline dimensions. Content analysis and scoring of these domains by the appraisal tools was evaluated.

RESULTS

Twenty-four different appraisal tools of practice guidelines were identified. None scored the evidence base of the clinical content of guidelines. Four tools scored all the guideline dimensions. The Cluzeau instrument is the only one of these four that has been validated. Of the three instruments based on the Cluzeau instrument, the AGREE instrument is the only validated instrument that uses a numerical scale.

CONCLUSIONS

Being a simplified version of the Cluzeau instrument, the AGREE instrument has the most potential to serve as a basis for the development of an appraisal tool for clinical pathways. However, important limitations will have to be dealt with when developing such a tool.

All dysuria is local. A cost-effectiveness model for designing site-specific management algorithms

OBJECTIVE

To determine the cost-effectiveness of management strategies for dysuria in different office settings.

DESIGN

Decision and cost-effectiveness analyses, assuming the payer's perspective. Data on disease prevalence, test characteristics, treatment efficacy, and adverse effects were drawn from the English language literature using medline searches and bibliographies.

SETTING

Hypothetical primary care practice.

PATIENTS

Otherwise healthy, nonpregnant women with symptoms of dysuria, urgency, and frequency.

INTERVENTIONS

All reasonable combinations of urinalysis, urine culture, pelvic examination, chlamydia and gonorrhea cultures, and empiric treatment with trimethoprim-sulfamethoxazole.

RESULTS

The cost-effectiveness of strategies varied substantially among different patient settings. In all settings, empiric trimethoprim-sulfamethoxazole for all patients was least expensive and least effective. Most testing increased both cost and effectiveness. Compared to empiric antibiotics, performing pelvic examination and urine culture for women with normal urinalyses had a marginal cost-effectiveness ratio of $4 to $32 per symptom-day avoided (SDA). Adding urine culture for patients with pyuria had a marginal cost of $34 to $107 per SDA, which fell to $40/SDA when the prevalence of resistance to trimethoprim-sulfamethoxazole exceeded 40%. Pelvic examination and urine culture for all patients regardless of urinalysis results achieved the greatest benefit but at the highest cost (>$300 per SDA).

CONCLUSIONS

In otherwise healthy women with symptoms of dysuria and no vaginal complaints, performing pelvic exam and urine culture based on urinalysis offers a reasonable alternative to empiric therapy. Other testing may be warranted, depending on antibiotic resistance and the value of avoiding a day of dysuria.

Advances in biomedical informatics for the management of cancer

Increased access to health care, and advances in education and technology have resulted in a larger proportion of the population having longer life expectancy. The strong correlation between age and cancer has resulted in a major healthcare problem for this century, and until recently cancer has defied any long-lasting cure. However, progress, especially in the field of biomedical informatics, promises a successful prediction and possibly a permanent cure for cancer within the next two decades. Biomedical informatics-with its roots in computer science, biomedical engineering, biostatistics, and mathematics-helps to bring the patient closer to the physician, facilitates access to specialist information and knowledge bases across the world, and makes it possible to identify genetic expression profiles for malignant or cancerous cells. This paper reviews the new research findings in biomedical informatics, working toward the ultimate goal of successfully predicting cancer, solving complex problems in prevention and treatment of cancer, and perhaps completely curing the scourge of cancer.

Evidence-based practice for mere mortals: the role of informatics and health services research

The poor translation of evidence into practice is a well-known problem. Hopes are high that information technology can help make evidence-based practice feasible for mere mortal physicians. In this paper, we draw upon the methods and perspectives of clinical practice, medical informatics, and health services research to analyze the gap between evidence and action, and to argue that computing systems for bridging this gap should incorporate both informatics and health services research expertise. We discuss 2 illustrative systems--trial banks and a web-based system to develop and disseminate evidence-based guidelines (alchemist)--and conclude with a research and training agenda.

Quality and methods of developing practice guidelines

BACKGROUND

It is not known whether there are differences in the quality and recommendations between evidence-based (EB) and consensus-based (CB) guidelines. We used breast cancer guidelines as a case study to assess for these differences.

METHODS

Five different instruments to evaluate the quality of guidelines were identified by a literature search. We also searched MEDLINE and the Internet to locate 8 breast cancer guidelines. These guidelines were classified in three categories: evidence based, consensus based and consensus based with no explicit consideration of evidence (CB-EB). Each guideline was evaluated by three of the authors using each of the instruments. For each guideline we assessed the agreement among 14 decision points which were selected from the NCCN (National Cancer Comprehensive Network) guidelines algorithm. For each decision point we recorded the level of the quality of the information used to support it. A regression analysis was performed to assess if the percentage of high quality evidence used in the guidelines development was related to the overall quality of the guidelines.

RESULTS

Three guidelines were classified as EB, three as CB-EB and two as CB. The EB guidelines scored better than CB, with the CB-EB scoring in the middle among all instruments for guidelines quality assessment. No major disagreement in recommendations was detected among the guidelines regardless of the method used for development, but the EB guidelines had a better agreement with the benchmark guideline for any decision point. When the source of evidence used to support decision were of high quality, we found a higher level of full agreement among the guidelines' recommendations. Up to 94% of variation in the quality score among guidelines could be explained by the quality of evidence used for guidelines development.

CONCLUSION

EB guidelines have a better quality than CB guidelines and CB-EB guidelines. Explicit use of high quality evidence can lead to a better agreement among recommendations. However, no major disagreement among guidelines was noted regardless of the method for their development.

Cognitive psychological studies of representation and use of clinical practice guidelines

Clinical practice guidelines provide a means to enhance physician performance. This investigation was undertaken in an attempt to understand the nature of impact of guideline use on physician performance. We investigated the impact of (a) algorithmic-based and (b) text-based practice guidelines on clinical decision-making by physicians at varying levels of expertise. Data were collected using clinical scenarios and a think-aloud paradigm, both with (primed) and without (spontaneous) the use of the guidelines. The two guidelines used in the study were management of diabetes and screening for thyroid disease. The results show that guidelines were used as reminders for both experts and non-experts. Guidelines acted as an educational tool for non-experts by assisting in knowledge reorganization, particularly for the non-experts. Text and algorithmic guideline formats were both useful to physician performance depending on the purpose of use: solving clinical problems or learning. These results provide insights into how guidelines can be fine-tuned for different users and for different purposes. Empirical research, coupled with design principles from the cognitive sciences, can form an essential component of guideline design and development.

Publishing web-based guidelines using interactive decision models

Commonly used methods for guideline development and dissemination do not enable developers to tailor guidelines systematically to specific patient populations and update guidelines easily. We developed a web-based system, ALCHEMIST, that uses decision models and automatically creates evidence-based guidelines that can be disseminated, tailored and updated over the web. Our objective was to demonstrate the use of this system with clinical scenarios that provide challenges for guideline development. We used the ALCHEMIST system to develop guidelines for three clinical scenarios: (1) Chlamydia screening for adolescent women, (2) antiarrhythmic therapy for the prevention of sudden cardiac death; and (3) genetic testing for the BRCA breast-cancer mutation. ALCHEMIST uses information extracted directly from the decision model, combined with the additional information from the author of the decision model, to generate global guidelines. ALCHEMIST generated electronic web-based guidelines for each of the three scenarios. Using ALCHEMIST, we demonstrate that tailoring a guideline for a population at high-risk for Chlamydia changes the recommended policy for control of Chlamydia from contact tracing of reported cases to a population-based screening programme. We used ALCHEMIST to incorporate new evidence about the effectiveness of implantable cardioverter defibrillators (ICD) and demonstrate that the cost-effectiveness of use of ICDs improves from $74 400 per quality-adjusted life year (QALY) gained to $34 500 per QALY gained. Finally, we demonstrate how a clinician could use ALCHEMIST to incorporate a woman's utilities for relevant health states and thereby develop patient-specific recommendations for BRCA testing; the patient-specific recommendation improved quality-adjusted life expectancy by 37 days. The ALCHEMIST system enables guideline developers to publish both a guideline and an interactive decision model on the web. This web-based tool enables guideline developers to tailor guidelines systematically, to update guidelines easily, and to make the underlying evidence and analysis transparent for users.