Lack of preregistered analysis plans allows unacceptable data mining for and selective reporting of consensus in Delphi studies

Establishing open science research priorities in health psychology: a research prioritisation Delphi exercise

OBJECTIVE

Research on Open Science practices in Health Psychology is lacking. This meta-research study aimed to identify research question priorities and obtain consensus on the Top 5 prioritised research questions for Open Science in Health Psychology.

METHODS AND MEASURES

An international Delphi consensus study was conducted. Twenty-three experts in Open Science and Health Psychology within the European Health Psychology Society (EHPS) suggested research question priorities to create a 'long-list' of items (Phase 1). Forty-three EHPS members rated the importance of these items, ranked their top five and suggested their own additional items (Phase 2). Twenty-four EHPS members received feedback on Phase 2 responses and then re-rated and re-ranked their top five research questions (Phase 3).

RESULTS

The top five ranked research question priorities were: 1. 'To what extent are Open Science behaviours currently practised in Health Psychology?', 2. 'How can we maximise the usefulness of Open Data and Open Code resources?', 3. 'How can Open Data be increased within Health Psychology?', 4. 'What interventions are effective for increasing the adoption of Open Science in Health Psychology?' and 5. 'How can we increase free Open Access publishing in Health Psychology?'.

CONCLUSION

Funding and resources should prioritise the research questions identified here.

Neural mechanisms underlying attentional bias modification in fibromyalgia patients: a double-blind ERP study

There is a growing interest in the potential benefits of attentional bias modification (ABM) training in chronic pain patients. However, studies examining the effectiveness of ABM programs in fibromyalgia patients have demonstrated inconclusive effects on both behavioral indices and clinical symptoms. Additionally, underlying neural dynamics of ABM effects could yield new insights but remain yet unexplored. Current study, therefore, aims to investigate the effects of ABM training on known neural electrophysiological indicators of attentional bias to pain (P2, N2a). Thirty-two fibromyalgia patients were enrolled and randomly assigned to an ABM training (N = 16) or control (N = 16) condition (2 weeks duration). Within the ABM training condition participants performed five sessions consisting of a modified version of the dot-probe task in which patients were trained to avoid facial pain expressions, whereas in the control group participants performed five sessions consisting of a standard version of the dot-probe task. Potential ABM training effects were evaluated by comparing a single pre- and post-treatment session, in which event-related potentials (ERPs) were recorded in response to both facial expressions and target stimuli. Furthermore, patients filled out a series of self-report questionnaires assessing anxiety, depression, pain-related worrying, fear of pain, fatigue and pain status. After training, results indicated an overall reduction of the amplitude of the P2 component followed by an enhancement of N2a amplitude for the ABM condition compared to control condition. In addition, scores on anxiety and depression decreased in patients assigned to the training condition. However, we found no effects derived from the training on pain-related and fatigue status. Present study offers new insights related to the possible neural mechanisms underlying the effect of ABM training in fibromyalgia. Clinical trial (TRN: NCT05905159) retrospectively registered (30/05/2023).

Panel stacking is a threat to consensus statement validity

Consensus statements can be very influential in medicine and public health. Some of these statements use systematic evidence synthesis but others fail on this front. Many consensus statements use panels of experts to deduce perceived consensus through Delphi processes. We argue that stacking of panel members toward one particular position or narrative is a major threat, especially in absence of systematic evidence review. Stacking may involve financial conflicts of interest, but nonfinancial conflicts of strong advocacy can also cause major bias. Given their emerging importance, we describe here how such consensus statements may be misleading, by analyzing in depth a recent high-impact Delphi consensus statement on COVID-19 recommendations as a case example. We demonstrate that many of the selected panel members and at least 35% of the core panel members had advocated toward COVID-19 elimination (Zero-COVID) during the pandemic and were leading members of aggressive advocacy groups. These advocacy conflicts were not declared in the Delphi consensus publication, with rare exceptions. Therefore, we propose that consensus statements should always require rigorous evidence synthesis and maximal transparency on potential biases toward advocacy or lobbyist groups to be valid. While advocacy can have many important functions, its biased impact on consensus panels should be carefully avoided.

Evidence-informed and consensus-based statements about SAFEty of Physical Agent Modalities Practice in physiotherapy and rehabilitation medicine (SAFE PAMP): a national Delphi of healthcare scientific societies

OBJECTIVE

A shared consensus on the safety about physical agent modalities (PAMs) practice in physiotherapy and rehabilitation is lacking. We aimed to develop evidence-informed and consensus-based statements about the safety of PAMs.

STUDY DESIGN AND SETTING

A RAND-modified Delphi Rounds' survey was used to reach a consensus. We established a steering committee of the Italian Association of Physiotherapy (Associazione Italiana di Fisioterapia) to identify areas and questions for developing statements about the safety of the most commonly used PAMs in physiotherapy and rehabilitation. We invited 28 National Scientific and Technical Societies, including forensics and lay members, as a multidisciplinary and multiprofessional panel of experts to evaluate the nine proposed statements and formulate additional inputs. The level of agreement was measured using a 9-point Likert scale, with consensus in the Delphi Rounds assessed using the rating proportion with a threshold of 75%.

RESULTS

Overall, 17 (61%) out of 28 scientific and technical societies participated, involving their most representative members. The panel of experts mainly consisted of clinicians (88%) with expertise in musculoskeletal (47%), pelvic floor (24%), neurological (18%) and lymphatic (6%) disorders with a median experience of 30 years (IQR=17-36). Two Delphi rounds were necessary to reach a consensus. The final approved criteria list comprised nine statements about the safety of nine PAMs (ie, electrical stimulation neuromodulation, extracorporeal shock wave therapy, laser therapy, electromagnetic therapy, diathermy, hot thermal agents, cryotherapy and therapeutic ultrasound) in adult patients with a general note about populations subgroups.

CONCLUSIONS

The resulting consensus-based statements inform patients, healthcare professionals and policy-makers regarding the safe application of PAMs in physiotherapy and rehabilitation practice. Future research is needed to extend this consensus on paediatric and frail populations, such as immunocompromised patients.

Rating versus ranking in a Delphi survey: a randomized controlled trial

BACKGROUND

The Delphi technique has steeply grown in popularity in health research as a structured approach to group communication process. Rating and ranking are two different procedures commonly used to quantify participants' opinions in Delphi surveys. We explored the influence of using a rating or ranking approach on item prioritization (main outcome), questionnaire completion time, and evaluation of task difficulty in a Delphi survey aimed at identifying priorities for the organization of primary cardiovascular care.

METHODS

A randomized controlled parallel group trial was embedded in a three-round online Delphi survey. After an "open" first round, primary care patients, trained patient partners, and primary care clinicians from seven primary care practices were allocated 1:1 to a rating or ranking assessment group for the remainder of the study by stratified permuted block randomization, with strata based on participants' gender and status. Agreement on item prioritization between the experimental groups was measured by calculating Krippendorff's alpha reliability coefficient on the aggregate rank order of items in each group after the final round. Self-reported ease or difficulty with the assessment task was measured with the Single Ease Question.

RESULTS

Thirty-six panelists (13 clinic patients, 7 patient partners, 16 clinicians; 60% females) were randomized to the rating (n = 18) or ranking (n = 18) group, with 30 (83%) completing all rounds. Both groups identified the same highest priorities from a set of 41 items, but significant discrepancies were found as early as the seventh top item. There was moderately strong agreement between the priority ordering of top items common to both groups (Krippendorff's alpha = 0.811, 95% CI = 0.669-0.920). A 9-min mean difference to complete the third-round questionnaire in favor of the rating group failed to achieve statistical significance (p = 0.053). Ranking was perceived as more difficult (p < 0.001).

CONCLUSIONS

A rating or ranking procedure led to modestly similar item prioritization in a Delphi survey, but ranking was more difficult. This study should be replicated with a larger number of participants and with variations in the ranking and rating procedures.

TRIAL REGISTRATION

Not applicable.

How to reach agreement: the impact of different analytical approaches to Delphi process results in core outcomes set development

BACKGROUND

Core outcomes sets are increasingly used to define research outcomes that are most important for a condition. Different consensus methods are used in the development of core outcomes sets; the most common is the Delphi process. Delphi methodology is increasingly standardised for core outcomes set development, but uncertainties remain. We aimed to empirically test how the use of different summary statistics and consensus criteria impact Delphi process results.

METHODS

Results from two unrelated child health Delphi processes were analysed. Outcomes were ranked by mean, median, or rate of exceedance, and then pairwise comparisons were undertaken to analyse whether the rankings were similar. The correlation coefficient for each comparison was calculated, and Bland-Altman plots produced. Youden's index was used to assess how well the outcomes ranked highest by each summary statistic matched the final core outcomes sets. Consensus criteria identified in a review of published Delphi processes were applied to the results of the two child-health Delphi processes. The size of the consensus sets produced by different criteria was compared, and Youden's index was used to assess how well the outcomes that met different criteria matched the final core outcomes sets.

RESULTS

Pairwise comparisons of different summary statistics produced similar correlation coefficients. Bland-Altman plots showed that comparisons involving ranked medians had wider variation in the ranking. No difference in Youden's index for the summary statistics was found. Different consensus criteria produced widely different sets of consensus outcomes (range: 5-44 included outcomes). They also showed differing abilities to identify core outcomes (Youden's index range: 0.32-0.92). The choice of consensus criteria had a large impact on Delphi results.

DISCUSSION

The use of different summary statistics is unlikely to affect how outcomes are ranked during a Delphi process: mean, median, and rates of exceedance produce similar results. Different consensus criteria have a large impact on resultant consensus outcomes and potentially on subsequent core outcomes sets: our results confirm the importance of adhering to pre-specified consensus criteria.

Use of Delphi in health sciences research: A narrative review

The use of the Delphi technique is prevalent across health sciences research, and it is used to identify priorities, reach consensus on issues of importance and establish clinical guidelines. Thus, as a form of expert opinion research, it can address fundamental questions present in healthcare. However, there is little guidance on how to conduct them, resulting in heterogenous Delphi studies and methodological confusion. Therefore, the purpose of this review is to introduce the use of the Delphi method, assess the application of the Delphi technique within health sciences research, discuss areas of methodological uncertainty and propose recommendations. Advantages of the use of Delphi include anonymity, controlled feedback, flexibility for the choice of statistical analysis, and the ability to gather participants from geographically diverse areas. Areas of methodological uncertainty worthy of further discussion broadly include experts and data management. For experts, the definition and number of participants remain issues of contention, while there are ongoing difficulties with expert selection and retention. For data management, there are issues with data collection, defining consensus and methods of data analysis, such as percent agreement, central tendency, measures of dispersion, and inferential statistics. Overall, the use of Delphi addresses important issues present in health sciences research, but methodological issues remain. It is likely that the aggregation of future Delphi studies will eventually pave the way for more comprehensive reporting guidelines and subsequent methodological clarity.

Existing guidance on reporting of consensus methodology: a systematic review to inform ACCORD guideline development

OBJECTIVE

To identify evidence on the reporting quality of consensus methodology and to select potential checklist items for the ACcurate COnsensus Reporting Document (ACCORD) project to develop a consensus reporting guideline.

DESIGN

Systematic review.

DATA SOURCES

Embase, MEDLINE, Web of Science, PubMed, Cochrane Library, Emcare, Academic Search Premier and PsycINFO from inception until 7 January 2022.

ELIGIBILITY CRITERIA

Studies, reviews and published guidance addressing the reporting quality of consensus methodology for improvement of health outcomes in biomedicine or clinical practice. Reports of studies using or describing consensus methods but not commenting on their reporting quality were excluded. No language restrictions were applied.

DATA EXTRACTION AND SYNTHESIS

Screening and data extraction of eligible studies were carried out independently by two authors. Reporting quality items addressed by the studies were synthesised narratively.

RESULTS

Eighteen studies were included: five systematic reviews, four narrative reviews, three research papers, three conference abstracts, two research guidance papers and one protocol. The majority of studies indicated that the quality of reporting of consensus methodology could be improved. Commonly addressed items were: consensus panel composition; definition of consensus and the threshold for achieving consensus. Items least addressed were: public patient involvement (PPI); the role of the steering committee, chair, cochair; conflict of interest of panellists and funding. Data extracted from included studies revealed additional items that were not captured in the data extraction form such as justification of deviation from the protocol or incentives to encourage panellist response.

CONCLUSION

The results of this systematic review confirmed the need for a reporting checklist for consensus methodology and provided a range of potential checklist items to report. The next step in the ACCORD project builds on this systematic review and focuses on reaching consensus on these items to develop the reporting guideline.

PROTOCOL REGISTRATION

https://osf.io/2rzm9.

ACCORD guideline for reporting consensus-based methods in biomedical research and clinical practice: a study protocol

BACKGROUND

Structured, systematic methods to formulate consensus recommendations, such as the Delphi process or nominal group technique, among others, provide the opportunity to harness the knowledge of experts to support clinical decision making in areas of uncertainty. They are widely used in biomedical research, in particular where disease characteristics or resource limitations mean that high-quality evidence generation is difficult. However, poor reporting of methods used to reach a consensus - for example, not clearly explaining the definition of consensus, or not stating how consensus group panellists were selected - can potentially undermine confidence in this type of research and hinder reproducibility. Our objective is therefore to systematically develop a reporting guideline to help the biomedical research and clinical practice community describe the methods or techniques used to reach consensus in a complete, transparent, and consistent manner.

METHODS

The ACCORD (ACcurate COnsensus Reporting Document) project will take place in five stages and follow the EQUATOR Network guidance for the development of reporting guidelines. In Stage 1, a multidisciplinary Steering Committee has been established to lead and coordinate the guideline development process. In Stage 2, a systematic literature review will identify evidence on the quality of the reporting of consensus methodology, to obtain potential items for a reporting checklist. In Stage 3, Delphi methodology will be used to reach consensus regarding the checklist items, first among the Steering Committee, and then among a broader Delphi panel comprising participants with a range of expertise, including patient representatives. In Stage 4, the reporting guideline will be finalised in a consensus meeting, along with the production of an Explanation and Elaboration (E&E) document. In Stage 5, we plan to publish the reporting guideline and E&E document in open-access journals, supported by presentations at appropriate events. Dissemination of the reporting guideline, including a website linked to social media channels, is crucial for the document to be implemented in practice.

DISCUSSION

The ACCORD reporting guideline will provide a set of minimum items that should be reported about methods used to achieve consensus, including approaches ranging from simple unstructured opinion gatherings to highly structured processes.

Developing a medication adherence technologies repository: proposed structure and protocol for an online real-time Delphi study

INTRODUCTION

An online interactive repository of available medication adherence technologies may facilitate their selection and adoption by different stakeholders. Developing a repository is among the main objectives of the European Network to Advance Best practices and technoLogy on medication adherencE (ENABLE) COST Action (CA19132). However, meeting the needs of diverse stakeholders requires careful consideration of the repository structure.

METHODS AND ANALYSIS

A real-time online Delphi study by stakeholders from 39 countries with research, practice, policy, patient representation and technology development backgrounds will be conducted. Eleven ENABLE members from 9 European countries formed an interdisciplinary steering committee to develop the repository structure, prepare study protocol and perform it. Definitions of medication adherence technologies and their attributes were developed iteratively through literature review, discussions within the steering committee and ENABLE Action members, following ontology development recommendations. Three domains (product and provider information (D1), medication adherence descriptors (D2) and evaluation and implementation (D3)) branching in 13 attribute groups are proposed: product and provider information, target use scenarios, target health conditions, medication regimen, medication adherence management components, monitoring/measurement methods and targets, intervention modes of delivery, target behaviour determinants, behaviour change techniques, intervention providers, intervention settings, quality indicators and implementation indicators. Stakeholders will evaluate the proposed definition and attributes' relevance, clarity and completeness and have multiple opportunities to reconsider their evaluations based on aggregated feedback in real-time. Data collection will stop when the predetermined response rate will be achieved. We will quantify agreement and perform analyses of process indicators on the whole sample and per stakeholder group.

ETHICS AND DISSEMINATION

Ethical approval for the COST ENABLE activities was granted by the Malaga Regional Research Ethics Committee. The Delphi protocol was considered compliant regarding data protection and security by the Data Protection Officer from University of Basel. Findings from the Delphi study will form the basis for the ENABLE repository structure and related activities.

EXpert consensus On Diaphragm UltraSonography in the critically ill (EXODUS): a Delphi consensus statement on the measurement of diaphragm ultrasound-derived parameters in a critical care setting

Background

Diaphragm ultrasonography is rapidly evolving in both critical care and research. Nevertheless, methodologically robust guidelines on its methodology and acquiring expertise do not, or only partially, exist. Therefore, we set out to provide consensus-based statements towards a universal measurement protocol for diaphragm ultrasonography and establish key areas for research.

Methods

To formulate a robust expert consensus statement, between November 2020 and May 2021, a two-round, anonymous and online survey-based Delphi study among experts in the field was performed. Based on the literature review, the following domains were chosen: “Anatomy and physiology”, “Transducer Settings”, “Ventilator Impact”, “Learning and expertise”, “Daily practice” and “Future directions”. Agreement of ≥ 68% (≥ 10 panelists) was needed to reach consensus on a question.

Results

Of 18 panelists invited, 14 agreed to participate in the survey. After two rounds, the survey included 117 questions of which 42 questions were designed to collect arguments and opinions and 75 questions aimed at reaching consensus. Of these, 46 (61%) consensus was reached. In both rounds, the response rate was 100%. Among others, there was agreement on measuring thickness between the pleura and peritoneum, using > 10% decrease in thickness as cut-off for atrophy and using 40 examinations as minimum training to use diaphragm ultrasonography in clinical practice. In addition, key areas for research were established.

Conclusion

This expert consensus statement presents the first set of consensus-based statements on diaphragm ultrasonography methodology. They serve to ensure high-quality and homogenous measurements in daily clinical practice and in research. In addition, important gaps in current knowledge and thereby key areas for research are established.

Trial registration The study was pre-registered on the Open Science Framework with registration digital object identifier https://doi.org/10.17605/OSF.IO/HM8UG.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13054-022-03975-5.

A Delphi-method-based consensus guideline for definition of treatment-resistant depression for clinical trials

Criteria for treatment-resistant depression (TRD) and partially responsive depression (PRD) as subtypes of major depressive disorder (MDD) are not unequivocally defined. In the present document we used a Delphi-method-based consensus approach to define TRD and PRD and to serve as operational criteria for future clinical studies, especially if conducted for regulatory purposes. We reviewed the literature and brought together a group of international experts (including clinicians, academics, researchers, employees of pharmaceutical companies, regulatory bodies representatives, and one person with lived experience) to evaluate the state-of-the-art and main controversies regarding the current classification. We then provided recommendations on how to design clinical trials, and on how to guide research in unmet needs and knowledge gaps. This report will feed into one of the main objectives of the EUropean Patient-cEntric clinicAl tRial pLatforms, Innovative Medicines Initiative (EU-PEARL, IMI) MDD project, to design a protocol for platform trials of new medications for TRD/PRD.

Current decision support tools fail to agree or predict therapeutic decisions in a single cohort of unruptured intracranial aneurysms

BACKGROUND

There is limited evidence to direct the management of unruptured intracranial aneurysms. Models extrapolated from existing data have been proposed to guide treatment recommendations. The aim of this study is to assess whether a consensus-based treatment score (UIATS) or rupture rate estimation model (PHASES) can be used to benchmark UK multi-disciplinary team (MDT) practice.

METHODS

Prospective data was collected on a consecutive series of all patients with unruptured intracranial aneurysms (UIAs) presenting to a major UK neurovascular centre between 2012 and 2015. The agreement between the UIATS and PHASES scores, and their sensitivity and specificity in predicting the real-world MDT outcome were calculated and compared.

RESULTS

A total of 366 patients (456 aneurysms) were included in the analysis. The agreement between UIATS and MDT recommendation was low (weighted kappa 0.26 [95% CI 0.19, 0.32]); sensitivity and specificity were also low at 36% and 52% respectively. Groups that the MDT allocated to treatment, equipoise or no treatment had significantly different PHASES scores (p = 0.004). There was no significant difference between the two scores when predicting patients for whom MDT outcome was to recommend aneurysm treatment, but the UIATS score was superior in predicting patients who received an MDT recommendation of 'treatment-equipoise', or 'not-for-treatment' (AUC of 0.73 compared to 0.59 for PHASES).

CONCLUSIONS

The models studied failed to agree with the consensus view of multi-disciplinary team in a major neurovascular centre. We conclude that decision support tools such as the UIATS and PHASES scores should not be blindly introduced in respective institutions without prior internal validation, as they may not represent the local reality.

Barriers to appropriate prescribing in older adults with multimorbidity: A modified Delphi study

OBJECTIVES

We aimed to understand the barriers experienced by physicians when prescribing for older adults with multimorbidity in specialist outpatient clinics in Singapore.

METHODS

A modified Delphi study was conducted via email with 20 panel experts in the field of geriatric medicine. Barriers identified from an earlier scoping review were presented as statements to the panel.

RESULTS

Eleven barrier statements reached consensus with high importance according to the Delphi panel. Of these statements, seven (64%) belong to the domain of Environmental context and resources in the Theoretical Domains Framework (TDF), while the remaining barriers belong to the domains of skills, knowledge, intentions, and professional/social role and identity. The barriers are further linked to intervention functions in the Behaviour Change Wheel (BCW).

CONCLUSION

Linking the TDF domains to intervention functions revealed strategic directions for the development of an intervention to address the barriers and optimize prescribing.

Delphi methodology in healthcare research: How to decide its appropriateness

The Delphi technique is a systematic process of forecasting using the collective opinion of panel members. The structured method of developing consensus among panel members using Delphi methodology has gained acceptance in diverse fields of medicine. The Delphi methods assumed a pivotal role in the last few decades to develop best practice guidance using collective intelligence where research is limited, ethically/logistically difficult or evidence is conflicting. However, the attempts to assess the quality standard of Delphi studies have reported significant variance, and details of the process followed are usually unclear. We recommend systematic quality tools for evaluation of Delphi methodology; identification of problem area of research, selection of panel, anonymity of panelists, controlled feedback, iterative Delphi rounds, consensus criteria, analysis of consensus, closing criteria, and stability of the results. Based on these nine qualitative evaluation points, we assessed the quality of Delphi studies in the medical field related to coronavirus disease 2019. There was inconsistency in reporting vital elements of Delphi methods such as identification of panel members, defining consensus, closing criteria for rounds, and presenting the results. We propose our evaluation points for researchers, medical journal editorial boards, and reviewers to evaluate the quality of the Delphi methods in healthcare research.

Delphi methodology in healthcare research: How to decide its appropriateness

The Delphi technique is a systematic process of forecasting using the collective opinion of panel members. The structured method of developing consensus among panel members using Delphi methodology has gained acceptance in diverse fields of medicine. The Delphi methods assumed a pivotal role in the last few decades to develop best practice guidance using collective intelligence where research is limited, ethically/logistically difficult or evidence is conflicting. However, the attempts to assess the quality standard of Delphi studies have reported significant variance, and details of the process followed are usually unclear. We recommend systematic quality tools for evaluation of Delphi methodology; identification of problem area of research, selection of panel, anonymity of panelists, controlled feedback, iterative Delphi rounds, consensus criteria, analysis of consensus, closing criteria, and stability of the results. Based on these nine qualitative evaluation points, we assessed the quality of Delphi studies in the medical field related to coronavirus disease 2019. There was inconsistency in reporting vital elements of Delphi methods such as identification of panel members, defining consensus, closing criteria for rounds, and presenting the results. We propose our evaluation points for researchers, medical journal editorial boards, and reviewers to evaluate the quality of the Delphi methods in healthcare research.

Importance of Internal Variability in Clinical Trials of Cardiovascular Disease

A well conducted randomised controlled trial (RCT) is extremely important in the field of cardiovascular medicine. At the same time, it is equally important to understand the strengths and limitations of any RCT, and internal variability is a concept in clinical trials that is poorly understood. Variability in a clinical trial may be introduced at an individual level or during measurement, sampling, or conduct of the trial. It is not the same as internal validity, which is a broader concept of accuracy; to be valid, a study should minimise variability and have sound methodology. There are various steps that may be followed to minimise the internal variability in a clinical trial. One aspect of great importance is the adjudication process, which should be done meticulously and is often a step that is overlooked. It is important to standardise each step as much as possible, to ensure consistency and reduce noise at all levels. The concepts discussed in this review may serve as a roadmap to limit the influence of internal variability and maximise internal validity of RCT results.

Defining major trauma: a Delphi study

Introduction

Retrospective trauma scores are often used to categorise trauma, however, they have little utility in the prehospital or hyper-acute setting and do not define major trauma to non-specialists. This study employed a Delphi process in order to gauge degrees of consensus/disagreement amongst expert panel members to define major trauma.

Method

A two round modified Delphi technique was used to explore subject-expert consensus and identify variables to define major trauma through systematically collating questionnaire responses.

After initial descriptive analysis of variables, Kruskal-Wallis tests were used to determine statistically significant differences (p < 0.05) in response to the Delphi statements between professional groups. A hierarchical cluster analysis was undertaken to identify patterns of similarity/difference of response.

A grounded theory approach to qualitative analysis of data allowed for potentially multiple iterations of the Delphi process to be influenced by identified themes.

Results

Of 55 expert panel members invited to participate, round 1 had 43 participants (Doctor n = 20, Paramedic n = 20, Nurse n = 5, other n = 2). No consistent patterns of opinion emerged with regards to professional group. Cluster analysis identified three patterns of similar responses and coded as trauma minimisers, the middle ground and the risk averse. Round 2 had 35 respondents with minimum change in opinion between rounds.

Consensus of > 70% was achieved on many variables which included the identification of life/limb threatening injuries, deranged physiology, need for intensive care interventions and that extremes of age need special consideration. It was also acknowledged that retrospective injury severity scoring has a role to play but is not the only method of defining major trauma. Various factors had a majority of agreement/disagreement but did not meet the pre-set criteria of 70% agreement. These included the topics of burns, spinal immobilisation and whether a major trauma centre is the only place where major trauma can be managed.

Conclusion

Based upon the output of this Delphi study, major trauma may be defined as: “Significant injury or injuries that have potential to be life-threatening or life-changing sustained from either high energy mechanisms or low energy mechanisms in those rendered vulnerable by extremes of age”.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13049-021-00870-w.

An Expert Consensus Study for Informed Consent in Primary Breast Augmentation Surgery

BACKGROUND

What constitutes adequate information for decision-making and informed consent is a practical question appropriately answered with deference to expertise.

OBJECTIVES

The aims of this study were: (1) to establish consensus on a procedure-specific core information set of essential informed consent information by relevant medical experts for primary breast augmentation surgery; and (2) to define from the clinical perspective the data source and imminence elements of evidence-based risk communication.

METHODS

The study followed a modified Delphi expert consensus model. Active members of The Aesthetic Society were identified as the relevant clinician experts and were recruited by email. Survey round 1 was informed by a scoping review of the relevant scientific and gray literature. Round 2 was informed by the initial survey round. Consensus was defined a priori as a 75% majority rating.

RESULTS

Expert consensus of essential information was achieved for 16 risk items, 1 risk factor, and 8 expectations-including benefits and burdens-along with clarification of clinically appropriate options to present to all patients considering primary implant-based breast augmentation surgery. A basic, procedure-specific, structure for evidence-based risk data is also described.

CONCLUSIONS

This paper reports results for the first phase of a larger pilot study aiming to develop a patient decision aid to replace traditional informed consent documents for the specified procedure. Implications for practice are encouraging in terms of reducing unwanted variation in disclosure practices and information overload.

Evaluating the efficacy of an attention modification program for patients with fibromyalgia: a randomized controlled trial

Persons with chronic musculoskeletal pain may be hypervigilant for pain-related cues which, paradoxically, may be maintaining their pain. Several randomized controlled trials have assessed whether a modified dot-probe protocol (ie, attention bias modification [ABM]) reduces chronic pain- and pain-related symptoms in persons with several diagnoses, including fibromyalgia. Scalability and economic efficiency potentiates the appeal of ABM protocols; however, research results have been mixed, with only some studies evidencing significant symptom gains from ABM and some evidencing gains for the control group. The current randomized controlled trial sought to replicate and extend previous ABM research using idiosyncratic word stimuli and a 1-month follow-up. Participants included treatment-seeking adult women (n = 117) with fibromyalgia who were randomly assigned to a standard (ie, control) or active (ie, ABM) condition. The protocol was delivered online and involved twice-weekly 15-minute sessions, for 4 weeks, with questionnaires completed at baseline, posttreatment, and 1-month follow-up. Symptom reports were analysed with mixed hierarchical modelling. There was no evidence of differences between the control and ABM groups. Both groups had small significant (Ps < 0.05) improvements in pain experiences at posttreatment, but not at follow-up (Ps > 0.05). There were no significant changes for either group on measures of anxiety sensitivity, illness/injury sensitivity, pain-related fear, pain-related anxiety, or attentional biases (Ps > 0.05). The current findings add to the emerging and mixed literature regarding ABM for pain by demonstrating that ABM produces no substantive improvements in pain or pain-related constructs in a large sample of patients with fibromyalgia.

Inconsistency and low transparency were found between core outcome set protocol and full text publication: a comparative study

OBJECTIVES

The objective of the study was to assess inconsistencies between individual protocols and associated full-text publications in the development of core outcome sets (COSs).

STUDY DESIGN AND SETTING

Protocols and subsequent full-text publications were retrieved by searching the following electronic databases: PubMed, Embase, Web of Science, and the Core Outcome Measures in Effectiveness Trials database from inception to October 1, 2019. We summarized changes in the general and methodological characteristics by comparing the protocols with the full-text publications and reported change as information frequency and proportion.

RESULTS

A total of 24 protocols and 32 corresponding full-text publications that encompassed 14 study topics were identified from databases. In the identified initial list of outcomes, five COSs (20.8%) changed the included study type, none of which explained the reasons for these changes. In addition, eight COSs showed inconsistencies between the protocols and full-text publications in the searched databases, of which, only two studies explained the reasons for these changes. Compared with the protocols, three COSs changed the number of Delphi rounds, eight COSs changed the participants (stakeholder groups), and three COSs changed the consensus definition of the Delphi survey. Only two COSs explained the reason for changing the number of Delphi rounds, and none of the studies explained why the participants changed. For the face-to-face consensus meeting, we found that nine COSs changed the participants and none explained the reasons for these changes.

CONCLUSION

Our study found many inconsistencies between protocols and the full-text publications concerning COS development. These inconsistencies related to the included study types, databases searched, Delphi surveys, and face-to-face consensus meetings. As it is necessary to publish protocols before developing COSs, transparency regarding any changes to the methods is needed.

How to design a pre-specified statistical analysis approach to limit p-hacking in clinical trials: the Pre-SPEC framework

Results from clinical trials can be susceptible to bias if investigators choose their analysis approach after seeing trial data, as this can allow them to perform multiple analyses and then choose the method that provides the most favourable result (commonly referred to as 'p-hacking'). Pre-specification of the planned analysis approach is essential to help reduce such bias, as it ensures analytical methods are chosen in advance of seeing the trial data. For this reason, guidelines such as SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) and ICH-E9 (International Conference for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use) require the statistical methods for a trial's primary outcome be pre-specified in the trial protocol. However, pre-specification is only effective if done in a way that does not allow p-hacking. For example, investigators may pre-specify a certain statistical method such as multiple imputation, but give little detail on how it will be implemented. Because there are many different ways to perform multiple imputation, this approach to pre-specification is ineffective, as it still allows investigators to analyse the data in different ways before deciding on a final approach. In this article, we describe a five-point framework (the Pre-SPEC framework) for designing a pre-specified analysis approach that does not allow p-hacking. This framework was designed based on the principles in the SPIRIT and ICH-E9 guidelines and is intended to be used in conjunction with these guidelines to help investigators design the statistical analysis strategy for the trial's primary outcome in the trial protocol.

Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response

Background

A proper application of the Delphi technique is essential for obtaining valid research results. Medical researchers regularly use Delphi studies, but reports often lack detailed information on methodology and controlled feedback: in the medical literature, papers focusing on Delphi methodology issues are rare. Since the introduction of electronic surveys, details on response times remain scarce. We aim to bridge a number of gaps by providing a real world example covering methodological choices and response times in detail.

Methods

The objective of our e(lectronic)-Delphi study was to determine minimum standards for emergency departments (EDs) in the Netherlands. We opted for a two-part design with explicit decision rules. Part 1 focused on gathering and defining items; Part 2 addressed the main research question using an online survey tool. A two-person consensus rule was applied throughout: even after consensus on specific items was reached, panellists could reopen the discussion as long as at least two panellists argued similarly. Per round, the number of reminders sent and individual response times were noted. We also recorded the methodological considerations and evaluations made by the research team prior to as well as during the study.

Results

The study was performed in eight rounds and an additional confirmation round. Response rates were 100% in all rounds, resulting in 100% consensus in Part 1 and 96% consensus in Part 2. Our decision rules proved to be stable and easily applicable. Items with negative advice required more rounds before consensus was reached. Response delays were mostly due to late starts, but once panellists started, they nearly always finished the questionnaire on the same day. Reminders often yielded rapid responses. Intra-individual differences in response time were large, but quick responders remained quick.

Conclusions

We advise those considering Delphi study to follow the CREDES guideline, consider a two-part design, invest in personal commitment of the panellists, set clear decision rules, use a consistent lay-out and send out your reminders early. Adopting this overall approach may assist researchers in future Delphi studies and may help to improve the quality of Delphi designs in terms of improved rigor and higher response rates.

Abstracts presented at the European Association for Osseointegration (EAO) Congresses: Publication fate and discrepancies with full-length articles

OBJECTIVES

To investigate the full publication proportion (FPP) of abstracts presented at the 2010 and 2011 EAO Congresses, analyse the discrepancies between abstracts and their full publications, and explore potential predictors of FPP and discrepancies.

METHODS

Abstracts presented at the 2010 and 2011 EAO Congresses were retrieved. Associated full publications were identified by searching PubMed, Embase and Google Scholar. Discrepancies between abstracts and full publications were identified, classified and evaluated using a discrepancy score. The Kaplan-Meier survival analysis was used to describe cumulative FPP over time. Predictors for FPP and the discrepancy score were analysed using cox regression modelling and a linear regression model, respectively.

RESULTS

850 abstracts were included. The overall FPP was 36.4% with a median time lapse of 12 months. Higher FPP were significantly associated with oral presentation (HR=2.33; 95% CI: 1.68 to 3.22; p<0.001), multiple affiliations (HR =1.32; 95% CI: 1.00 to 1.73; p=0.048) and presence of statistical tests (HR =1.78; 95% CI: 1.36 to 2.32; p<0.001). 91.3% pairs had at least one minor change from the abstract and 70.9% had at least one major change. Greater discrepancy score was significantly associated with longer time lapse (B=0.06; 95% CI: 0.04 to 0.08; p<0.001) and being clinical research (B=1.30; 95% CI: 0.52 to 2.08; p=0.001).

CONCLUSIONS

Thirty-six percent of abstracts presented at the EAO Congresses were published. Among these, more than two-thirds showed at least one major change in their full publications. Abstracts presented in oral implantology conferences should not be relied upon to inform practice.

Comparison of different rating scales for the use in Delphi studies: different scales lead to different consensus and show different test-retest reliability

Background

Consensus-orientated Delphi studies are increasingly used in various areas of medical research using a variety of different rating scales and criteria for reaching consensus. We explored the influence of using three different rating scales and different consensus criteria on the results for reaching consensus and assessed the test-retest reliability of these scales within a study aimed at identification of global treatment goals for total knee arthroplasty (TKA).

Methods

We conducted a two-stage study consisting of two surveys and consecutively included patients scheduled for TKA from five German hospitals. Patients were asked to rate 19 potential treatment goals on different rating scales (three-point, five-point, nine-point). Surveys were conducted within a 2 week period prior to TKA, order of questions (scales and treatment goals) was randomized.

Results

Eighty patients (mean age 68 ± 10 years; 70% females) completed both surveys. Different rating scales (three-point, five-point and nine-point rating scale) lead to different consensus despite moderate to high correlation between rating scales (r = 0.65 to 0.74). Final consensus was highly influenced by the choice of rating scale with 14 (three-point), 6 (five-point), 15 (nine-point) out of 19 treatment goals reaching the pre-defined 75% consensus threshold. The number of goals reaching consensus also highly varied between rating scales for other consensus thresholds. Overall, concordance differed between the three-point (percent agreement [p] = 88.5%, weighted kappa [k] = 0.63), five-point (p = 75.3%, k = 0.47) and nine-point scale (p = 67.8%, k = 0.78).

Conclusion

This study provides evidence that consensus depends on the rating scale and consensus threshold within one population. The test-retest reliability of the three rating scales investigated differs substantially between individual treatment goals. This variation in reliability can become a potential source of bias in consensus studies. In our setting aimed at capturing patients’ treatment goals for TKA, the three-point scale proves to be the most reasonable choice, as its translation into the clinical context is the most straightforward among the scales. Researchers conducting Delphi studies should be aware that final consensus is substantially influenced by the choice of rating scale and consensus criteria.

Practical Considerations in Using Online Modified-Delphi Approaches to Engage Patients and Other Stakeholders in Clinical Practice Guideline Development

Patients and caregivers are increasingly recognized as key stakeholders in developing clinical practice guidelines (CPGs). Online engagement approaches offer the promise of a rigorous, scalable, and convenient engagement method. This paper illustrates how an online modified-Delphi approach could be used to engage patients, caregivers, and other stakeholder in CPG development. It provides practical guidance for conducting online modified-Delphi panels that covers (1) joint development of the engagement approach with relevant stakeholders, (2) adaptation of methods used by experts in guideline development, (3) pilot testing, (4) participant recruitment, (5) determining panel size and composition, (6) building participant capacity, (7) facilitation of two-way interaction, (8) continuous engagement and retention of participants, (9) rigorous data analysis, (10) evaluation of engagement activities, and (11) result dissemination. The paper is based on a recently completed study about engaging individuals with Duchenne muscular dystrophy (DMD) and their caregivers in determining the patient-centeredness of DMD care guidelines.

Delphi consensus reached to produce a decision tool for SelecTing Approaches for Rapid Reviews (STARR)

OBJECTIVES

There are many rapid review methods; however, there is little pragmatic guidance on which methods to select. This study aimed to reach consensus among international rapid review experts outlining areas to consider when selecting approaches for rapid reviews.

STUDY DESIGN AND SETTING

A two-round modified online Delphi survey was conducted between May and July 2018. Participants were asked to rank the importance of a predefined list of 19 items. A consensus definition of at least 70% agreement for each item was decided a priori.

RESULTS

Thirty experts from ten countries participated in round 1 and 24 in round 2. During round 1, consensus was reached on all items. One additional item on quality assessment was suggested by respondents and comments suggested wording changes to improve clarity and understanding of the tool. Respondents in the second round indicated a high level of importance and all 20 items achieved consensus. These items addressed interaction with commissioners, scoping and searching the evidence-base, data extraction and synthesis methods, and reporting of rapid review methods.

CONCLUSION

International consensus was reached to produce the SelecTing Approaches for Rapid Reviews (STARR) decision tool for planning rapid reviews and will lead to improved shared understanding between review teams and review commissioners.

Delphi procedure in core outcome set development: rating scale and consensus criteria determined outcome selection

OBJECTIVE

The objective of this study was to compare two different rating scales within one Delphi study for defining consensus in core outcome set development and to explore the influence of consensus criteria on the outcome selection.

STUDY DESIGN AND SETTING

Randomized controlled parallel group trial with 1:1 allocation within the first Delphi round of the Core Outcome Set in the Incontinence-Associated Dermatitis project. Outcomes were rated on a three-point or nine-point Likert scale. Decisions about which outcomes to retain were determined by commonly used consensus criteria (i.e., [combinations of] proportions with restricted ranges, central tendency within a specific range, and decrease in variance).

RESULTS

Fifty-seven participants (group 1 = 28, group 2 = 29) rated 58 outcomes. The use of the nine-point scale resulted in almost twice as many outcomes being rated as "critical" compared with the three-point scale (24 vs. 13). Stricter criteria and combining criteria led to less outcomes being identified as "critical".

CONCLUSION

The format of rating scales in Delphi studies for core outcome set development and the definition of the consensus criteria influence outcome selection. The use of the nine-point scale might be recommended to inform the consensus process for a subsequent rating or face-to-face meeting. The three-point scale might be preferred when determining final consensus.

Consensus Building in OMERACT: Recommendations for Use of the Delphi for Core Outcome Set Development

Objective.

Developing international consensus on outcome measures for clinical trials is challenging. The following paper will review consensus building in Outcome Measures in Rheumatology (OMERACT), with a focus on the Delphi.

Methods.

Based on the literature and feedback from delegates at OMERACT 2018, a set of recommendations is provided in the form of the OMERACT Delphi Consensus Checklist.

Results.

The OMERACT delegates generally supported the use of the checklist as a guide. The checklist provides guidance for clearly outlining the multiple aspects of the Delphi process.

Conclusion.

OMERACT is deeply committed to consensus building and these recommendations should be considered a work in progress.

Pre-specification of statistical analysis approaches in published clinical trial protocols was inadequate

OBJECTIVES

Results from randomized trials can depend on the statistical analysis approach used. It is important to prespecify the analysis approach in the trial protocol to avoid selective reporting of analyses based on those which provide the most favourable results. We undertook a review of published trial protocols to assess how often the statistical analysis of the primary outcome was adequately prespecified.

METHODS

We searched protocols of randomized trials indexed in PubMed in November 2016. We identified whether the following aspects of the statistical analysis approach for the primary outcome were adequately prespecified: (1) analysis population; (2) analysis model; (3) use of covariates; and (4) method of handling missing data.

RESULTS

We identified 99 eligible protocols. Very few protocols adequately prespecified the analysis population (8/99, 8%), analysis model (27/99, 27%), covariates (40/99, 40%), or approach to handling missing data (10/99, 10%). Most protocols did not adequately predefine any of these four aspects of their statistical analysis approach (39%) or predefined only one aspect (36%). No protocols adequately predefined all four aspects of the analysis.

CONCLUSION

The statistical analysis approach is rarely prespecified in published trial protocols. This may allow selective reporting of results based on different analyses.