An experimental study of the influence of individual participant characteristics on formal consensus development

Variability of physicians' thresholds for neuroimaging in children with recurrent headache

Background

We sought to determine the extent to which physicians agree about the appropriate decision threshold for recommending magnetic resonance imaging in a clinical practice guideline for children with recurrent headache.

Methods

We surveyed attending physicians in Canada practicing in community pediatrics, child neurology, pediatric radiology, and pediatric neurosurgery. For children in each of six risk categories, physicians were asked to determine whether they would recommend for or against routine magnetic resonance imaging of the brain in a clinical practice guideline for children with recurrent headache.

Results

Completed surveys were returned by 114 physicians. The proportion recommending routine neuroimaging for each risk group was 100% (50% risk), 99% (10% risk), 93% (4% risk), 54% (1% risk), 25% (0.4% risk), 4% (0.01% risk). Community pediatricians, physicians in practice >15 years, and physicians who believed they ordered neuroimaging less often than peers were less likely to recommend neuroimaging for the 1% risk group (all p < 0.05).

Conclusions

There is no consensus among pediatric specialists regarding the appropriate decision threshold for neuroimaging in a clinical practice guideline for children with recurrent headache. Because of the impact that individual threshold preferences may have on guidelines, these findings support the need for careful composition of guideline committees and consideration of the role of patient and family preferences. Our findings also support the need for transparency in guidelines regarding how evidence was translated into recommendations and how conflicts were resolved.

Human computation as a new method for evidence-based knowledge transfer in Web-based guideline development groups: proof of concept randomized controlled trial

BACKGROUND

Guideline developers use different consensus methods to develop evidence-based clinical practice guidelines. Previous research suggests that existing guideline development techniques are subject to methodological problems and are logistically demanding. Guideline developers welcome new methods that facilitate a methodologically sound decision-making process. Systems that aggregate knowledge while participants play a game are one class of human computation applications. Researchers have already proven that these games with a purpose are effective in building common sense knowledge databases.

OBJECTIVE

We aimed to evaluate the feasibility of a new consensus method based on human computation techniques compared to an informal face-to-face consensus method.

METHODS

We set up a randomized design to study 2 different methods for guideline development within a group of advanced students completing a master of nursing and obstetrics. Students who participated in the trial were enrolled in an evidence-based health care course. We compared the Web-based method of human-based computation (HC) with an informal face-to-face consensus method (IC). We used 4 clinical scenarios of lower back pain as the subject of the consensus process. These scenarios concerned the following topics: (1) medical imaging, (2) therapeutic options, (3) drugs use, and (4) sick leave. Outcomes were expressed as the amount of group (dis)agreement and the concordance of answers with clinical evidence. We estimated within-group and between-group effect sizes by calculating Cohen's d. We calculated within-group effect sizes as the absolute difference between the outcome value at round 3 and the baseline outcome value, divided by the pooled standard deviation. We calculated between-group effect sizes as the absolute difference between the mean change in outcome value across rounds in HC and the mean change in outcome value across rounds in IC, divided by the pooled standard deviation. We analyzed statistical significance of within-group changes between round 1 and round 3 using the Wilcoxon signed rank test. We assessed the differences between the HC and IC groups using Mann-Whitney U tests. We used a Bonferroni adjusted alpha level of .025 in all statistical tests. We performed a thematic analysis to explore participants' arguments during group discussion. Participants completed a satisfaction survey at the end of the consensus process.

RESULTS

Of the 135 students completing a master of nursing and obstetrics, 120 participated in the experiment. We formed 8 HC groups (n=64) and 7 IC groups (n=56). The between-group comparison demonstrated that the human computation groups obtained a greater improvement in evidence scores compared to the IC groups, although the difference was not statistically significant. The between-group effect size was 0.56 (P=.30) for the medical imaging scenario, 0.07 (P=.97) for the therapeutic options scenario, and 0.89 (P=.11) for the drug use scenario. We found no significant differences in improvement in the degree of agreement between HC and IC groups. Between-group comparisons revealed that the HC groups showed greater improvement in degree of agreement for the medical imaging scenario (d=0.46, P=.37) and the drug use scenario (d=0.31, P=.59). Very few evidence arguments (6%) were quoted during informal group discussions.

CONCLUSIONS

Overall, the use of the IC method was appropriate as long as the evidence supported participants' beliefs or usual practice, or when the availability of the evidence was sparse. However, when some controversy about the evidence existed, the HC method outperformed the IC method. The findings of our study illustrate the importance of the choice of the consensus method in guideline development. Human computation could be an acceptable methodology for guideline development specifically for scenarios in which the evidence shows no resonance with participants' beliefs. Future research is needed to confirm the results of this study and to establish practical significance in a controlled setting of multidisciplinary guideline panels during real-life guideline development.

Influence of expert clinical adviser characteristics on opinions about interventional procedures

Objectives:The aim of this study was to examine potential associations between the professional background and experience of expert clinicians and their opinions about the clinical utility of interventional procedures.

Methods:A retrospective survey of expert clinician characteristics and their opinions was conducted. Information was collected on expert clinical adviser self-declared “operator,” “researcher,” and conflict of interest status. Associations were sought between expert clinical adviser characteristics and their opinions on whether procedures were “established,” “efficacious,” and “safe.” The setting was the Interventional Procedures Programme of the UK's National Institute for Health and Clinical Excellence (NICE). A total of 598 expert clinician questionnaires relating to 182 different interventional procedures were analyzed.

Results:Expert clinical advisers with operative experience of procedures were significantly more likely to consider them as established (odds ratio [OR] 3.93; 95 percent confidence interval [CI], 2.43 to 6.36;p&lt; .001), efficacious (OR 1.76; 95 percent CI, 1.00 to 3.08;p= .049), and safe (OR 2.28; 95 percent CI, 1.43 to 3.65;p= .001). Once adjusted for other characteristics, there was no association between either researcher or conflict of interest status and opinions about the clinical utility of procedures.

Conclusions: Expert clinical advisers are an important source of information for decision makers producing guidance about the use of procedures, especially when published evidence is sparse or of poor quality. This study suggests that those who are operators, but not those who are researchers or declare a conflict of interest, are more likely to have a favorable opinion of a procedure's clinical utility. Use of expert clinical advisers with a variety of experience and backgrounds seems a reasonable approach to obtaining authoritative opinions about interventional procedures, to supplement and help interpret evidence from published data.

Diagnostic imaging practice guidelines for musculoskeletal complaints in adults--an evidence-based approach: introduction

PURPOSE

Imaging technology can improve patient outcomes by allowing greater precision in diagnosing and treating patients. However, there is evidence that overuse, underuse, and misuse of imaging services occur. The purpose of this project was to develop evidence-based diagnostic imaging practice guidelines for musculoskeletal complaints for use by doctors of chiropractic and other primary health care professionals.

METHODS

An electronic search of the English and French language literature (phase 1) was conducted on several databases. Cross references, and references provided by clinicians, were also used. Independent assessment of the quality of the citations used to support recommendations in the guidelines was performed using the QUADAS, the AGREE,and the SPREAD evaluation tools. A first draft of a diagnostic imaging practice guideline was produced, using the European Commission's Referral Guidelines for Imaging document as a template. Results were sent to 12 chiropractic specialists for a first external review. A modified Delphi process, including 149 international experts, was used to generate consensus on recommendations for diagnostic imaging studies. The reliability of proposed recommendations was further tested on field chiropractors and on a group of specialists both in chiropractic and in medicine in both Canada and the United States. All recommendations were graded according to the strength of the evidence.

RESULTS

The research procedure resulted in the recommendations for diagnostic imaging guidelines of adult extremity and spine disorders supported by more than 685 primary and secondary citations. High levels of agreement among Delphi panelists were reached for all proposed recommendations. Comments received by specialists were generally very favorable and reflected high levels of agreement with the proposed recommendations, perceived ease of use of guidelines, and implementation feasibility.

CONCLUSIONS

These evidence-based diagnostic imaging practice guidelines are intended to assist chiropractors and other primary care providers in decision making on the appropriate use of diagnostic imaging for specific clinical presentations. In all cases, the guidelines are intended to be used in conjunction with sound clinical judgment and experience. Application of these guidelines should help avoid unnecessary radiographs, increase examination precision,and decrease health care costs without compromising the quality of care. All guidelines are documents to be refined and modified regularly with new information and experience.