Mapping the nomenclature, methodology, and reporting of studies that review methods: a pilot methodological review

Evidence-informed guidelines in oral health: insights from a systematic survey

BACKGROUND

Oral diseases are a major global public health problem, impacting the quality of life of those affected. While consensus exists on the importance of high-quality, evidence-informed guidelines to inform practice and public health decisions in medicine, appropriate methodologies and standards are not commonly adhered to among producers of oral health guidelines. This study aimed to systematically identify organizations that develop evidence-informed guidelines in oral health globally and survey the methodological process followed to formulate recommendations.

METHODS

We searched numerous electronic databases, guideline repositories, and websites of guideline developers, scientific societies, and international organizations (January 2012-October 2023) to identify organizations that develop guidelines addressing any oral health topic and that explicitly declare the inclusion of research evidence in their development. Pairs of reviewers independently evaluated potentially eligible organizations according to predefined selection criteria and extracted data about the organization's characteristics, key features of their guidelines, and the process followed when formulating formal recommendations. Descriptive statistics were used to analyze and summarize data.

RESULTS

We included 46 organizations that developed evidence-informed guidelines in oral health. The organizations were mainly professional associations and scientific societies (67%), followed by governmental organizations (28%). In total, organizations produced 55 different guideline document types, most of them containing recommendations for clinical practice (77%). Panels were primarily composed of healthcare professionals (87%), followed by research methodologists (40%), policymakers (24%), and patient partners (18%). Most (60%) of the guidelines reported their funding source, but only one out of three (33%) included a conflict of interest (COI) policy management. The methodology used in the 55 guideline document types varied across the organizations, but only 19 (35%) contained formal recommendations. Half (51%) of the guideline documents referred to a methodology handbook, 46% suggested a structured approach or system for rating the certainty of the evidence and the strength of recommendations, and 37% mentioned using a framework to move from evidence to decisions, with the GRADE-EtD being the most widely used (27%).

CONCLUSION

Our findings underscore the need for alignment and standardization of both terminology and methodologies used in oral health guidelines with current international standards to formulate trustworthy recommendations.

The use of GRADE-CERQual in qualitative evidence synthesis: an evaluation of fidelity and reporting

BACKGROUND

GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative Research) is a methodological approach to systematically and transparently assess how much confidence decision makers can place in individual review findings from qualitative evidence syntheses. The number of reviews applying GRADE-CERQual is rapidly expanding in guideline and other decision-making contexts. The objectives of this evaluation were, firstly, to describe the uptake of GRADE-CERQual in qualitative evidence synthesis by review authors and, secondly, to assess both reporting of and fidelity to the approach.

METHODS

The evaluation had two parts. Part 1 was a citation analysis and descriptive overview of the literature citing GRADE-CERQual. Authors worked together to code and chart the citations, first by title and abstract and second by full text. Part 2 was an assessment and analysis of fidelity to, and reporting of, the GRADE-CERQual approach in included reviews. We developed fidelity and reporting questions and answers based on the most recent guidance for GRADE-CERQual and then used NVivo12 to document assessments in a spreadsheet and code full-text PDF articles for any concerns that had been identified. Our assessments were exported to Excel and we applied count formulae to explore patterns in the data. We employed a qualitative content analysis approach in NVivo12 to sub-coding all the data illustrating concerns for each reporting and fidelity criteria.

RESULTS

233 studies have applied the GRADE-CERQual approach, with most (n = 225, 96.5%) in the field of health research. Many studies (n = 97/233, 41.6%) were excluded from full fidelity and reporting assessment because they demonstrated a serious misapplication of GRADE-CERQual, for example interpreting it as a quality appraisal tool for primary studies or reviews. For the remaining studies that applied GRADE-CERQual to assess confidence in review findings, the main areas of reporting concern involved terminology, labelling and completeness. Fidelity concerns were identified in more than half of all studies assessed.

CONCLUSIONS

GRADE-CERQual is being used widely within qualitative evidence syntheses and there are common reporting and fidelity issues. Most of these are avoidable and we highlight these as gaps in knowledge and guidance for applying the GRADE-CERQual approach.

Using systems perspectives in evidence synthesis: A methodological mapping review

Reviewing complex interventions is challenging because they include many elements that can interact dynamically in a nonlinear manner. A systems perspective offers a way of thinking to help understand complex issues, but its application in evidence synthesis is not established. The aim of this project was to understand how and why systems perspectives have been applied in evidence synthesis. A methodological mapping review was conducted to identify papers using a systems perspective in evidence synthesis. A search was conducted in seven bibliographic databases and three search engines. A total of 101 papers (representing 98 reviews) met the eligibility criteria. Two categories of reviews were identified: (1) reviews using a "systems lens" to frame the topic, generate hypotheses, select studies, and guide the analysis and interpretation of findings (n = 76) and (2) reviews using systems methods to develop a systems model (n = 22). Several methods (e.g., systems dynamic modeling, soft systems approach) were identified, and they were used to identify, rank and select elements, analyze interactions, develop models, and forecast needs. The main reasons for using a systems perspective were to address complexity, view the problem as a whole, and understand the interrelationships between the elements. Several challenges for capturing the true nature and complexity of a problem were raised when performing these methods. This review is a useful starting point when designing evidence synthesis of complex interventions. It identifies different opportunities for applying a systems perspective in evidence synthesis, and highlights both commonplace and less familiar methods.

Globe and Adnexal Trauma Terminology Survey

Importance

Ocular trauma terminology should be periodically updated to enable comprehensive capturing and monitoring of ocular trauma in clinical and research settings.

Objective

To update terminology for globe and adnexal trauma.

Design, Setting, and Participants

A 2-round modified Delphi survey was conducted from January 1 to July 31, 2021, using an expert panel, including 69 ophthalmologists identified through their membership in ophthalmology (globe and adnexal trauma) societies. Consensus was defined as at least 67% expert agreement. A steering committee developed questions after identifying gaps in the current terminology via a targeted literature review. Round 1 sought consensus on existing and newly proposed terminology, and round 2 focused on unresolved questions from round 1. Experts included ophthalmologists who had managed, on average, 52 globe or adnexal trauma cases throughout their careers and/or published a total of 5 or more globe or adnexal trauma-related peer-reviewed articles.

Main Outcomes and Measures

Expert consensus on ocular and adnexal terms.

Results

A total of 69 experts participated in and completed round 1 of the survey. All 69 participants who completed round 1 were asked to complete round 2, and 58 responses were received. Consensus was reached for 18 of 25 questions (72%) in round 1 and 4 of 7 questions (57%) in round 2. Existing Birmingham Eye Trauma Terminology system terminology achieved consensus of 84% (58 of 69 experts) in round 1 and 97% (56 of 58 experts) in round 2. Experts agreed on the need for further refinement of the definition of zones of injury (55 of 69 [80%]), as the zone affected can have a substantial effect on visual and functional outcomes. There was consensus that the mechanism of injury (52 of 69 [75%]) and status of the lacrimal canaliculi (54 of 69 [78%]), nasolacrimal ducts (48 of 69 [69%]), lens (46 of 58 [80%]), retina (42 of 58 [73%]), and central and paracentral cornea (47 of 58 [81%]) be included in the revised terminology.

Conclusions and Relevance

There was consensus (defined as at least 67% expert agreement) on continued use of the existing Birmingham Eye Trauma Terminology system definitions and that additional terms are required to update the current ocular trauma terminology.

The use of cognitive task analysis in clinical and health services research — a systematic review

Background

At times, clinical case complexity and different types of uncertainty present challenges to less experienced clinicians or the naive application of clinical guidelines where this may not be appropriate. Cognitive task analysis (CTA) methods are used to elicit, document and transfer tacit knowledge about how experts make decisions.

Methods

We conducted a methodological review to describe the use of CTA methods in understanding expert clinical decision-making. We searched MEDLINE, EMBASE and PsycINFO from inception to 2019 for primary research studies which described the use of CTA methods to understand how qualified clinicians made clinical decisions in real-world clinical settings.

Results

We included 81 articles (80 unique studies) from 13 countries, published from 1993 to 2019, most commonly from surgical and critical care settings. The most common aims were to understand expert decision-making in particular clinical scenarios, using expert decision-making in the development of training programmes, understanding whether decision support tools were warranted and understanding procedural variability and error identification or reduction. Critical decision method (CDM) and CTA interviews were most frequently used, with hierarchical task analysis, task knowledge structures, think-aloud protocols and other methods less commonly used. Studies used interviews, observation, think-aloud exercises, surveys, focus groups and a range of more CTA-specific methodologies such as the systematic human error reduction and prediction approach. Researchers used CTA methods to investigate routine/typical (n = 64), challenging (n = 13) or more uncommon, rare events and anomalies (n = 3).

Conclusions

In conclusion, the elicitation of expert tacit knowledge using CTA has seen increasing use in clinical specialties working under challenging time pressures, complexity and uncertainty. CTA methods have great potential in the development, refinement, modification or adaptation of complex interventions, clinical protocols and practice guidelines.

Registration

PROSPERO ID CRD42019128418.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40814-022-01002-6.

Outcomes reported in randomised controlled trials of major depressive disorder in older adults: protocol for a methodological review

INTRODUCTION

Major depressive disorder (MDD or depression) is prevalent among adults aged 65 years and older. The effectiveness and safety of interventions used to treat depression is often assessed through randomised controlled trials (RCTs). However, heterogeneity in the selection, measurement and reporting of outcomes in RCTs renders comparisons between trial results, interpretability and generalisability of findings challenging. There is presently no core outcome set (COS) for use in RCTs that assess interventions for older adults with MDD. We will conduct a methodological review of the literature for outcomes reported in trials for adults 65 years and older with depression to assess the heterogeneity of outcome measures.

METHODS AND ANALYSIS

RCTs evaluating pharmacotherapy, psychotherapy, or any other treatment intervention for older adults with MDD published in the last 10 years will be located using electronic database searches (MEDLINE, Embase, PsycINFO and the Cochrane Central Register of Controlled Trials). Reviewers will conduct title and abstract screening, full-text screening and data extraction of trials eligible for inclusion independently and in duplicate. Outcomes will be synthesised and mapped to core outcome-domain frameworks. We will summarise characteristics associated with trials and outcomes.

ETHICS AND DISSEMINATION

We hope that findings from our methodological review will reduce variability in outcome selection, measurement and reporting and facilitate the development of a COS for older adults with MDD. Our review will also inform evidence synthesis efforts in identifying the best treatment practices for this clinical population. Ethics approval is not required, as this study is a literature review.

PROSPERO REGISTRATION NUMBER

CRD42021244753.

Evaluating the quality of reported outcomes for microsurgical TESE in men with non-obstructive azoospermia: A methodological analysis

BACKGROUND

Publications of microsurgical testicular sperm extraction (mTESE) techniques and outcomes are heterogeneous, which may limit creation of best surgical practices.

OBJECTIVE

To study the quality and heterogeneity of published mTESE outcomes via a methodological analysis.

MATERIALS/METHODS

A systematic methodological analysis of all published literature on the use of mTESE in men with non-obstructive azoospermia from 1999 to the July 2020 was performed. PubMed and MEDLINE searches were performed using the search terms "microdissection TESE OR microsurgical TESE." Publications were evaluated on their reporting of pre-operative factors, intraoperative techniques, surgical and clinical outcomes, and adverse events.

RESULTS

Fifty-five studies met inclusion criteria. Surgical technique and sperm retrieval rates were the most reported criteria. Reporting on the presence of an embryologist intraoperatively was observed in approximately 30% of articles, while other procedural details including method of sperm quantification, quantity retrieved, and number of cryopreserved vials were observed in fewer than 10% of articles. Clinical outcomes, including pregnancy rates and live birth rates, were reported in fewer than 40% of the articles. Fetal outcomes including fetal and neonatal anomalies were rarely reported. Fetal growth restriction, preterm delivery, small or large for gestational age, and NICU admissions were never reported.

CONCLUSION

There are inconsistencies in reporting quality of mTESE outcomes, specifically a lack of information on the quantity and quality of sperm retrieved, the role of embryology intraoperatively, and clinical outcomes, such as live birth rate and fetal anomalies. These gaps may guide development of standardized reporting guidelines to better assess and compare clinical outcomes across institutions and maintain focus on couples-centric fertility outcomes in future mTESE studies.

The Structure of Research Questions in Randomized Controlled Trials in the Rehabilitation Field: A Methodological Study

OBJECTIVE

The aim of this study is to assess whether and how the PICO (Population, Intervention, Comparator, and Outcomes) format is described to frame research questions in randomized controlled trials looking at effectiveness of rehabilitation interventions.

DESIGN

A methodological study was conducted. Randomized controlled trials in the rehabilitation field, published between July 1, 2019, and December 31, 2019, were included. The framing of the primary research question from each trial was evaluated.

RESULTS

A total of 97 randomized controlled trials were included in the analysis. The most frequent framing of the primary research question was as an "objective" statement (55%), and in 33% of the articles, this was stated as an "objective" together with a "hypothesis" description. All PICO elements were present in 55% of research questions, but only 49% have used the statement suggested by Cochrane.

CONCLUSION

The findings of this study suggest that a specific item about the "research question" and the rationale that drove the proposed design following the form suggested by Cochrane should be included in the RCT Rehabilitation Checklist.

Reporting of methodological studies in health research: a protocol for the development of the MethodologIcal STudy reportIng Checklist (MISTIC)

INTRODUCTION

Methodological studies (ie, studies that evaluate the design, conduct, analysis or reporting of other studies in health research) address various facets of health research including, for instance, data collection techniques, differences in approaches to analyses, reporting quality, adherence to guidelines or publication bias. As a result, methodological studies can help to identify knowledge gaps in the methodology of health research and strategies for improvement in research practices. Differences in methodological study names and a lack of reporting guidance contribute to lack of comparability across studies and difficulties in identifying relevant previous methodological studies. This paper outlines the methods we will use to develop an evidence-based tool-the MethodologIcal STudy reportIng Checklist-to harmonise naming conventions and improve the reporting of methodological studies.

METHODS AND ANALYSIS

We will search for methodological studies in the Cumulative Index to Nursing and Allied Health Literature, Cochrane Library, Embase, MEDLINE, Web of Science, check reference lists and contact experts in the field. We will extract and summarise data on the study names, design and reporting features of the included methodological studies. Consensus on study terms and recommended reporting items will be achieved via video conference meetings with a panel of experts including researchers who have published methodological studies.

ETHICS AND DISSEMINATION

The consensus study has been exempt from ethics review by the Hamilton Integrated Research Ethics Board. The results of the review and the reporting guideline will be disseminated in stakeholder meetings, conferences, peer-reviewed publications, in requests to journal editors (to endorse or make the guideline a requirement for authors), and on the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network and reporting guideline websites.

REGISTRATION

We have registered the development of the reporting guideline with the EQUATOR Network and publicly posted this project on the Open Science Framework (www.osf.io/9hgbq).

A tutorial on methodological studies: the what, when, how and why

BACKGROUND

Methodological studies - studies that evaluate the design, analysis or reporting of other research-related reports - play an important role in health research. They help to highlight issues in the conduct of research with the aim of improving health research methodology, and ultimately reducing research waste.

MAIN BODY

We provide an overview of some of the key aspects of methodological studies such as what they are, and when, how and why they are done. We adopt a "frequently asked questions" format to facilitate reading this paper and provide multiple examples to help guide researchers interested in conducting methodological studies. Some of the topics addressed include: is it necessary to publish a study protocol? How to select relevant research reports and databases for a methodological study? What approaches to data extraction and statistical analysis should be considered when conducting a methodological study? What are potential threats to validity and is there a way to appraise the quality of methodological studies?

CONCLUSION

Appropriate reflection and application of basic principles of epidemiology and biostatistics are required in the design and analysis of methodological studies. This paper provides an introduction for further discussion about the conduct of methodological studies.