Are systematic reviews and meta-analyses still useful research? We are not sure

Can mindfulness-based interventions reduce PTSD symptoms? An umbrella review

Post-traumatic stress disorder (PTSD) is a debilitating, often chronic condition with substantial cross-national lifetime prevalence. Although mindfulness-based interventions (MBIs) may help reduce PTSD symptoms, efficacy results are inconsistent. Despite many systematic reviews (SRs) examining MBIs for PTSD, SR quality has been neither evaluated nor synthesized. We conducted an umbrella review to summarize and evaluate existing evidence regarding MBIs for PTSD, identifying 69 SRs (27 meta-analyses), consisting of 83 primary studies. Using AMSTAR2 (a valid SR quality assessment tool), we evaluated each SR on key domains relevant to methodological rigor and rated the confidence of inferences. Results found SRs were 65.2% non-rigorous, 27.5% likely rigorous, and 7.2% rigorous; common limitations included inadequate risk of bias assessment, extractions not completed in duplicate, and lack of pre-registration, highlighting the need for higher quality SRs. We then performed a meta-meta-analysis to estimate the efficacy of MBIs to reduce PTSD symptoms, yielding a medium effect size (SMD=0.41, p < .001), derived from 22 meta-analyses (with replicable data) and 35 unique articles. Analyses were consistent across control conditions and MBI type (first-generation/narrow [i.e., MBIs with well-established protocols]) versus broad (i.e., other MBI types), comparable with second-line treatments (e.g., pharmacotherapy). Findings were narratively synthesized; areas for methodological improvements in MBI research were identified.

Reporting Bias in Systematic Reviews and Meta-Analyses Related to the Treatment of Distal Radius Fractures: The Presence of Spin in the Abstract

BACKGROUND

Spin is a form of reporting bias which suggests a treatment is beneficial despite a statistically nonsignificant difference in outcomes. Our purpose was to define the prevalence of spin within the abstracts of distal radius fracture (DRF) systematic reviews (SRs) and meta-analyses (MA). We also sought to identify article characteristics that were more likely to contain spin.

METHODS

We performed a SR of multiple databases to identify DRF SRs and MAs. Articles were screened and analyzed by 3 reviewers. We recorded article and journal characteristics including adherence to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, funding disclosures, methodologic quality (AMSTAR 2 instrument), impact factor, and country of origin. Presence of the 9 most severe types of spin in abstracts were recorded. Unadjusted odds ratios (ORs) were calculated to analyze the association between article characteristics and the presence of spin.

RESULTS

A total of 112 articles were included. Spin was present in 46% of abstracts, with type 1 spin ("conclusions not supported by findings") most frequent (19%). Spin was present in 43% of abstracts in PRISMA-adhering journals compared to 49% in journals that did not (OR = 0.79, 95% confidence interval [CI] = 0.37-1.68). For articles originating from China, spin was present in 61% of abstracts compared to 39% of abstracts from other countries (OR = 2.55, 95% CI = 1.13-5.75).

CONCLUSIONS

In addition to low article quality, there are high rates of spin within the abstracts of SRs and MAs related to treatment of DRF. Articles within journals that adhere to PRISMA do not appear to contain less spin.

The Australian Traumatic Brain Injury Initiative: Statement of Working Principles and Rapid Review of Methods to Define Data Dictionaries for Neurological Conditions

The Australian Traumatic Brain Injury Initiative (AUS-TBI) aims to develop a health informatics approach to collect data predictive of outcomes for persons with moderate-severe TBI across Australia. Central to this approach is a data dictionary; however, no systematic reviews of methods to define and develop data dictionaries exist to-date. This rapid systematic review aimed to identify and characterize methods for designing data dictionaries to collect outcomes or variables in persons with neurological conditions. Database searches were conducted from inception through October 2021. Records were screened in two stages against set criteria to identify methods to define data dictionaries for neurological conditions (International Classification of Diseases, 11th Revision: 08, 22, and 23). Standardized data were extracted. Processes were checked at each stage by independent review of a random 25% of records. Consensus was reached through discussion where necessary. Thirty-nine initiatives were identified across 29 neurological conditions. No single established or recommended method for defining a data dictionary was identified. Nine initiatives conducted systematic reviews to collate information before implementing a consensus process. Thirty-seven initiatives consulted with end-users. Methods of consultation were "roundtable" discussion (n = 30); with facilitation (n = 16); that was iterative (n = 27); and frequently conducted in-person (n = 27). Researcher stakeholders were involved in all initiatives and clinicians in 25. Importantly, only six initiatives involved persons with lived experience of TBI and four involved carers. Methods for defining data dictionaries were variable and reporting is sparse. Our findings are instructive for AUS-TBI and can be used to further development of methods for defining data dictionaries.

Importance of Correctly Classifying Systematic Reviews and Meta-Analysis: A Letter to the Editor

The Journal of Immigrant and Minority Health recently published a review article by Aran et al. (2023) containing important findings about patterns of suicidality in Newcomers. Although the review provided a valuable contribution to the literature, there was an issue with misclassification of the type of review and analysis the authors conducted. In this letter, I make the distinction between the relevant types of review and analyses and emphasize the importance of correctly identifying systematic reviews and meta-analyses to help provide clarity in the ongoing debate about the value of systematic reviews and meta-analyses. I recommend Aran et al. (2023) re-evaluate the classification and presentation of their important research to avoid confusion and mitigate potential adverse impacts on the scientific community.

The need for robust critique of arts and health research: the treatment of the Gene Cohen et al. (2006) paper on singing, wellbeing and health in subsequent evidence reviews

BACKGROUND

This paper considers weaknesses in a study by Cohen et al. (2006) on the impacts of community singing on health. These include high demand characteristics, lack of attention to attrition, flawed statistical analysis, and measurement. Nevertheless, the study is uncritically cited, in evidence reviews, with findings taken at face value.

METHODS

Google Scholar, SCOPUS and BASE citation functions for Cohen et al. identified 32 evidence reviews in peer-reviewed journals. Eleven of these reviews, published between 2010 and 2023, focused on creative arts interventions.

RESULTS

We demonstrate limitations in the Cohen et al. research which undermine the conclusions they reach regarding the health benefits of group singing. Subsequent evidence reviews take the findings at face value and offer little critical commentary.

DISCUSSION

We consider what is needed to improve evidence reviews in the field of creative arts and health research.

CONCLUSIONS

A more robust approach is needed in reviewing research evidence in the field of arts and health. The Cohen et al. paper is not suitable for inclusion in future evidence reviews.

Guidance to best tools and practices for systematic reviews

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Guidance to best tools and practices for systematic reviews

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Guidance to best tools and practices for systematic reviews

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

The use of immunotherapies for recurrent miscarriage: An overview of systematic reviews and meta-analysis

Immunotherapies have been a treatment proposed for recurrent miscarriages (RMs). The use of immunotherapies remains not recommended in the management of couples with RM. This overview of systematic reviews and meta-analysis (SRs-MAs) aims to identify and evaluate the quality of SRs-MAs that studied the effectiveness of immunotherapies in the treatment of RM patients. SRs-MAs were searched in PubMed/Medline, Embase, and Web of Science. SRs-MAs were analyzed using AMSTAR-2, PRISMA 2020, Risk of Bias in Systematic (ROBIS), and Grading of Recommendations Assessment, Development, and Evaluation (GRADE) tools to evaluate the methodological quality, reporting quality, risk of bias, and evidence quality of included SRs-MAs, respectively. This review included 20 SRs-MAs that evaluated the following immunotherapies: intravenous immunoglobulin (13 publications), lymphocyte immunotherapy (6 publications), corticosteroids (3 publications), and lipid emulsion (1 publication). SRs-MAs were rated as high methodological, moderate, and critically low quality in 14 (70 %), 1 (5 %), and 5 (25 %) SRs-MAs and high reporting, moderate, and low quality in 13 (65 %), 4 (20 %), and 3 (5 %) SRs-MAs, respectively. The overall risk of bias revealed a low risk of bias for three-quarters of the SRs-MAs. Grading of Recommendations Assessment, Development, and Evaluation (GRADE) analysis resulted in 23 outcomes, of which 4, 3, 5, and 11 results were of high, moderate, low, and very low quality, respectively. An improvement has been observed over the past few years in the quality of systematic reviews (SR)-MAs that have investigated the efficacy of intravenous immunoglobulin, lymphocyte immunotherapy, lipid emulsion therapy, and corticosteroids as a therapy for RM.

Guidance to best tools and practices for systematic reviews

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Guidance to best tools and practices for systematic reviews

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Guidance to Best Tools and Practices for Systematic Reviews

» Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.» A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.» Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Current status of the reporting quality of abstracts in systematic reviews related to implant dentistry: a literature survey

The aim of this study was to assess the reporting quality of abstracts in systematic reviews (SRs) related to implant dentistry and to assess the possible factors associated with the reporting quality. Abstracts of SRs in the field of implant dentistry, published in the last 5 years, were searched. The reporting quality was assessed and scored using the PRISMA for Abstracts checklist (PRISMA-A). The overall PRISMA-A score (OPS) and relative score (OPS%) per review were calculated according to adherence to the criteria presented in the checklist. Multivariable linear regression was performed to identify possible factors associated with reporting quality. Overall, 310 SRs were eligible for this study. Based on the maximum PRISMA-A score (score of 12), the mean OPS was 6.5 and OPS% was 54.2%. The items 'title', 'objectives', and 'number of included studies' were those most frequently reported in the abstracts, while the items 'registration' and 'funding' were the least reported. According to multivariable linear regression, the geographical origin of the articles was the only factor associated with better quality of abstract reporting, with higher OPS for SRs from Europe when compared to North America (coefficient 0.73; P = 0.049). The reporting quality of abstracts in SRs related to implant dentistry is suboptimal and needs to be improved. Journals should encourage adherence to reporting checklists in SRs.

Importance of sequential methods in meta-analysis: implications for postoperative mortality, delirium, and stroke management

Trial sequential analysis is an adaptation of frequentist sequential methods that can be used to improve inferences from meta-analysis. Trial sequential analysis can help preserve type I and type II error rates at desired levels for analyses conducted before the required information size. Through three case studies recently published in the British Journal of Anaesthesia, we show how trial sequential analysis can inform the interpretation of meta-analyses. Limitations of trial sequential analysis, which also include those of the meta-analysis to which it is applied, must be carefully considered alongside its benefits.

Clinical outcomes in patients undergoing invasive mechanical ventilation using NAVA and other ventilation modes - A systematic review and meta-analysis

PURPOSE

Neurally adjusted ventilatory assist mode (NAVA) benefit in mechanical ventilation (MV) patients with regard to clinically outcomes is still uncertain. Recent randomized clinical trials (RCTs) have addressed this issue, making it important to assess the real impact of NAVA in relation to these outcomes.

MATERIALS AND METHODS

We performed a systematic review and meta-analysis of RCTs comparing NAVA ventilation mode versus the standard ventilation mode in critically ill adult patients admitted to the ICU with invasive MV. The main outcome was 28-days ventilatory free-days (VFD). Secondary outcomes were weaning failure, mortality, ICU and hospital length of stay and need for tracheostomy.

RESULTS

We included 5 RCTs (643 patients). The patients in the NAVA group had increased VFDs compared to the control group: mean difference (MD) 3.42 (95% CI 1.21 to 5.62, I² = 0%). NAVA and control groups did not differ in ICU mortality [OR 0.58 (95% CI 0.33 to 1.03), I2 = 41%]. NAVA mode was associated with a reduced incidence of weaning failure [OR 0.51 (95% CI 0.29 to 0.88), I² = 0%]. NAVA and control groups did not differ in the number of MV days: MD -1.9 days (95% CI -4.2 to 0.3, I² = 0%).

CONCLUSIONS

NAVA mode has a modest impact on MV-free days and weaning success, with no association with improvements in other relevant clinical outcomes.

Automating Quality Assessment of Medical Evidence in Systematic Reviews: Model Development and Validation Study

BACKGROUND

Assessment of the quality of medical evidence available on the web is a critical step in the preparation of systematic reviews. Existing tools that automate parts of this task validate the quality of individual studies but not of entire bodies of evidence and focus on a restricted set of quality criteria.

OBJECTIVE

We proposed a quality assessment task that provides an overall quality rating for each body of evidence (BoE), as well as finer-grained justification for different quality criteria according to the Grading of Recommendation, Assessment, Development, and Evaluation formalization framework. For this purpose, we constructed a new data set and developed a machine learning baseline system (EvidenceGRADEr).

METHODS

We algorithmically extracted quality-related data from all summaries of findings found in the Cochrane Database of Systematic Reviews. Each BoE was defined by a set of population, intervention, comparison, and outcome criteria and assigned a quality grade (high, moderate, low, or very low) together with quality criteria (justification) that influenced that decision. Different statistical data, metadata about the review, and parts of the review text were extracted as support for grading each BoE. After pruning the resulting data set with various quality checks, we used it to train several neural-model variants. The predictions were compared against the labels originally assigned by the authors of the systematic reviews.

RESULTS

Our quality assessment data set, Cochrane Database of Systematic Reviews Quality of Evidence, contains 13,440 instances, or BoEs labeled for quality, originating from 2252 systematic reviews published on the internet from 2002 to 2020. On the basis of a 10-fold cross-validation, the best neural binary classifiers for quality criteria detected risk of bias at 0.78 F₁ (P=.68; R=0.92) and imprecision at 0.75 F₁ (P=.66; R=0.86), while the performance on inconsistency, indirectness, and publication bias criteria was lower (F₁ in the range of 0.3-0.4). The prediction of the overall quality grade into 1 of the 4 levels resulted in 0.5 F₁. When casting the task as a binary problem by merging the Grading of Recommendation, Assessment, Development, and Evaluation classes (high+moderate vs low+very low-quality evidence), we attained 0.74 F₁. We also found that the results varied depending on the supporting information that is provided as an input to the models.

CONCLUSIONS

Different factors affect the quality of evidence in the context of systematic reviews of medical evidence. Some of these (risk of bias and imprecision) can be automated with reasonable accuracy. Other quality dimensions such as indirectness, inconsistency, and publication bias prove more challenging for machine learning, largely because they are much rarer. This technology could substantially reduce reviewer workload in the future and expedite quality assessment as part of evidence synthesis.

Evaluation of the use of GRADE in dentistry systematic reviews and its impact on conclusions: a protocol for a methodological study

BACKGROUND

There is a growing body of evidence of systematic reviews (SRs) with varying degrees of methodological quality. The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach allows SR authors to assess the certainty of the evidence they found and transparently relay their conclusions. As there appears to be infrequent utilization of GRADE in the field of dentistry, to identify the impact of GRADE, the aim of this study is to evaluate the use of GRADE in the dental literature and determine whether SRs that use GRADE differ from those that do not with respect to their conclusions.

METHODS/DESIGN

We will search Ovid MEDLINE for SRs published from 2016 to the present. We will conduct both screening and data extraction independently and in duplicate and use pre-piloted, standardized forms for data extraction. We will determine the frequency of the use of GRADE and the varying levels of certainty in the current literature and evaluate whether GRADE is being used appropriately. We will also evaluate whether SRs not using GRADE differ from those that use GRADE with regard to methodological quality. We will also determine whether the conclusions of SRs that do not use GRADE would change had GRADE been utilized. Additionally, we will evaluate whether SRs using GRADE are more likely to formulate appropriate conclusions compared to SRs that do not use it.

DISCUSSION

This study will investigate the frequency of GRADE assessments in dentistry SRs and the impact of GRADE assessments on the conclusions of a SR. It has important implications for both SR authors and users of this type of literature.

Efficacy of psychological interventions for PTSD in distinct populations - An evidence map of meta-analyses using the umbrella review methodology

We aimed at mapping the meta-analytic evidence base on the efficacy of psychological treatments for posttraumatic stress disorder (PTSD) in specific populations. We conducted a systematic search until January 2022 in MEDLINE, PsycINFO, PTSDpubs, Web of Science, and the Cochrane Database of Systematic Reviews for meta-analyses of randomized controlled trials. We contrasted all eligible meta-analyses irrespective of overlapping datasets to present a comprehensive overview of the state of research. Reporting quality was assessed using the AMSTAR 2 tool and certainty of evidence was assessed using established umbrella review criteria. Nine meta-analyses with distinct adult populations (51 unique trials) and four with children and adolescents (24 unique trials) were included. Reporting quality of meta-analyses was heterogeneous with risk of bias assessment being rated lowest. The certainty of evidence on the efficacy of psychological interventions for adult populations was thoroughly weak because of small samples and large heterogeneity. In war- and conflict-affected youth, the certainty of evidence was suggestive. Our review highlights the need to improve quality of meta-analyses on treatment efficacy for PTSD. More importantly, however, the findings demonstrate the need for new large-scale trials on the efficacy of treatments for PTSD in distinct populations in order to increase certainty of evidence and to identify potential differences in treatment responses.

Guidance to best tools and practices for systematic reviews1

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

The Need for Robust Critique of Arts and Health Research: Young People, Art Therapy and Mental Health

We describe work in progress to conduct a systematic review of research on effects of arts-based programs for mental health in young people. We are at the stage of searching for relevant studies through major databases and screening extant systematic reviews for additional research which meet our inclusion criteria. At this stage, however, concerns have arisen regarding both the quality of existing primary studies and of recently published systematic reviews in this area of arts and health. As a case in point, in this paper we focus on one research report on art therapy with adolescent girls and its inclusion in three systematic reviews. We demonstrate that the reviews fail to undertake a robust critique of the Bazargan and Pakdaman paper and that the paper and reviews are flawed. Drawing on recent criticisms of systematic reviewing, we consider the value of proceeding with our systematic review as initially planned.

Facilitating Open Science Practices for Research Syntheses: PreregRS Guides Preregistration

Researchers increasingly engage in adopting open science practices in the field of research syntheses, such as preregistration. Preregistration is a central open science practice in empirical research to enhance transparency in the research process and it gains steady adoption in the context of conducting research synthesis. From an interdisciplinary perspective, frameworks and particularly templates are lacking which support researchers preparing a preregistration. To this end, we introduce preregRS, a template to guide researchers across disciplines through the process of preregistering research syntheses. We utilized an R Markdown template file to provide a framework that structures the process of preparing a preregistration. Researchers can write up the preregistration using the template file similar to filling out a form, with the template providing additional hints and further information for the decisions along the framework. We integrated the R Markdown template in an R package for easy installation and use, but also provide a browser-based option for users granting low-barrier access. PreregRS constitutes a first step to facilitate and support preregistration with research syntheses for all disciplines. It further adds to establishing open science practices in conducting research syntheses. This article is protected by copyright. All rights reserved.

Quality appraisal of systematic reviews of interventions for children with cerebral palsy reveals critically low confidence

AIM

To evaluate the methodological quality of recent systematic reviews of interventions for children with cerebral palsy in order to determine the level of confidence in the reviews' conclusions.

METHOD

A comprehensive search of 22 databases identified eligible systematic reviews with and without meta-analysis published worldwide from 2015 to 2019. We independently extracted data and used A Measurement Tool to Assess Systematic Reviews-2 (AMSTAR-2) to appraise methodological quality.

RESULTS

Eighty-three systematic reviews met strict eligibility criteria. Most were from Europe and Latin America and reported on rehabilitative interventions. AMSTAR-2 appraisal found critically low confidence in 88% (n=73) because of multiple and varied deficiencies. Only 7% (n=6) had no AMSTAR-2 critical domain deficiency. The number of systematic reviews increased fivefold from 2015 to 2019; however, quality did not improve over time.

INTERPRETATION

Most of these systematic reviews are considered unreliable according to AMSTAR-2. Current recommendations for treating children with CP based on these flawed systematic reviews need re-evaluation. Findings are comparable to reports from other areas of medicine, despite the general perception that systematic reviews are high-level evidence. The required use of current widely accepted guidance for conducting and reporting systematic reviews by authors, peer reviewers, and editors is critical to ensure reliable, unbiased, and transparent systematic reviews. What this paper adds Confidence was critically low in the conclusions of 88% of systematic reviews about interventions for children with cerebral palsy (CP). Quality issues in the sample were not limited to systematic reviews of non-randomized trials, or to those about certain populations of CP or interventions. The inclusion of meta-analysis did not improve the level of confidence in these systematic reviews. Numbers of systematic reviews on this topic increased over the 5 search years but their methodological quality did not improve.

Exploring the drivers of malnutrition in West Africa from health and social science perspectives: A comparative methodological review

West Africa has a high burden of malnutrition and the drivers are often complex, highly context-specific, and cut across individual, social, political and environmental domains. Public health research most often considers immediate individual health and diet drivers, at the expense of wider considerations that may fall outside of a health agenda. The objective of this systematic mapping review is to map the broad drivers of malnutrition in West Africa, from public health and social science perspectives, and to evaluate the additional value of an interdisciplinary approach. Evidence was gathered from one public health (MEDLINE) and one social science (International Bibliography of Social Science) database using a detailed search syntax tailored to each disciplinary configuration. Literature was screened against pre-determined eligibility criteria and extracted from abstracts. Studies published in English or French between January 2010 and April 2018 were considered for inclusion. Driver categories (immediate, underlying and basic drivers) were coded against the UNICEF conceptual framework of malnutrition. A total of 358 studies were included; 237 were retrieved from the public health database and 124 from the social science database, 3 studies were included in both. The public health and social science literature document different drivers, with MEDLINE most often reporting immediate drivers of malnutrition and the International Bibliography of Social Science database reporting underlying and basic drivers. The combined literature offers more balanced representation across categories. An interdisciplinary approach proved successful in achieving complementarity in search results while upholding rigorous methods. We recommend that interdisciplinary approaches are utilised to bridge recognised gaps between defined disciplines.

Prevention and early intervention of depression in young people: an integrated narrative review of affective awareness and Ecological Momentary Assessment

Depression has a devastating impact on young people around the world. This impact is pervasive and long-long lasting, still causing havoc as young people transition into adulthood. Prevention and timely early intervention efforts are necessary to lessen the disease burden of depression in young people. There is some evidence for the effectiveness of existing prevention and early intervention programs when delivered early. However, there is no consensus on what the active ingredients of these programs are. Identifying these ingredients is important because focusing on those that have maximal benefit will help to minimise the costs and resourcing of (already intensive) therapeutic approaches. We explored whether, and in what circumstances, affective awareness (i.e., knowing how one feels) is a candidate for protecting against and decreasing vulnerability to depression in young people. We specifically looked at how Ecological Momentary Assessment methods (EMA) were used to measure and/or increase affective awareness. We integrated three streams of data, including narrative review findings, perspectives from young people and psychologists, and publicly available information online (e.g., discussion forums). Across data sources, we found converging evidence that affective awareness is important in the prevention and early intervention of depression although there were considerable gaps in knowledge. More work needs to be done with young people and their health care teams to build affective awareness in the right way, at the right time, with individual differences in mind. The identification of how EMA can be best incorporated into young people's lives to facilitate these outcomes is also needed.

Publication of meta-analyses in sleep medicine: a scoping review

STUDY OBJECTIVES

Sleep research has grown substantially in recent decades, producing a large amount of data and an increasing number of meta-analyses. This study sought to establish the volume of meta-analyses in this area and assess how this level of material has developed over time.

METHODS

A bibliographic search of the Web of Science database was conducted (1945-2019). The total number of articles and the total number of meta-analyses were extracted for both sleep medicine and a combination of 6 other medical specialties (cardiology, neurology, psychiatry, pulmonology, otorhinolaryngology, and pediatrics).

RESULTS

A total of 262,384 articles and 1,152 meta-analyses related to sleep medicine were identified. Considering the whole period under analysis, meta-analyses represented 0.44% of the total number of sleep medicine-related articles. Throughout this period, the proportion of meta-analyses published has been increasing in both sleep medicine and the other fields, but it is greater in the other fields. In 2019, meta-analyses in sleep medicine represented 1.10% of the publication output in this area but represented 1.62% of the other areas. However, sleep medicine's growth rate has been consistently higher than in the other fields. The United States, China, and the United Kingdom have been the top meta-analysis producers.

CONCLUSIONS

Meta-analyses in sleep medicine are underused. As a recent medical field, sleep medicine has more potential to grow and is likely to grow faster than other fields. Researchers should be encouraged to perform and publish meta-analyses on sleep medicine, as long as the analyses are reasonable and feasible from methodological, statistical. and practical perspectives.

Challenges to Evidence Synthesis and Identification of Data Gaps in Human Biomonitoring

The increasing number of human biomonitoring (HBM) studies undertaken in recent decades has brought to light the need to harmonise procedures along all phases of the study, including sampling, data collection and analytical methods to allow data comparability. The first steps towards harmonisation are the identification and collation of HBM methodological information of existing studies and data gaps. Systematic literature reviews and meta-analyses have been traditionally put at the top of the hierarchy of evidence, being increasingly applied to map available evidence on health risks linked to exposure to chemicals. However, these methods mainly capture peer-reviewed articles, failing to comprehensively identify other important, unpublished sources of information that are pivotal to gather a complete map of the produced evidence in the area of HBM. Within the framework of the European Human Biomonitoring Initiative (HBM4EU) initiative-a project that joins 30 countries, 29 from Europe plus Israel, the European Environment Agency and the European Commission-a comprehensive work of data triangulation has been made to identify existing HBM studies and data gaps across countries within the consortium. The use of documentary analysis together with an up-to-date platform to fulfil this need and its implications for research and practice are discussed.

Electronic consenting for conducting research remotely: A review of current practice and key recommendations for using e-consenting

BACKGROUND

Electronic approaches are becoming more widely used to obtain informed consent for research participation. Electronic consent (e-consent) provides an accessible and versatile approach to the consenting process, which can be enhanced with audio-visual and interactive features to improve participant engagement and comprehension of study procedures. Best practice guidance underpinned by ethical principles is required to ensure effective implementation of e-consent for use in research.

AIM

To identify the key considerations for successful and ethical implementation of e-consent in the recruitment of participants to research projects which are conducted remotely.

METHODS

Electronic database searches of CINAHL, Medline, Embase, DARE, HTA, PubMed, the Cochrane Library, Scopus, Web of Science, NHS Evidence, and hand-searches of reference lists were performed. Primary research studies of adult (≥ 18 years old) research participants using e-consent, published in English language, peer-reviewed journals between 2010-2020 were eligible for inclusion.

RESULTS

Of the initial 665 identified studies, 18 met the inclusion criteria: 6 cohort studies, 5 qualitative studies, 4 randomised control trials, 2 mixed-methods studies and one case-control study. Critical appraisal of included studies using Critical Appraisal Skills Program (CASP) tools suggested a low to moderate risk of bias in most studies (n = 15). Key practice recommendations for researchers using e-consent were identified around five primary themes: 1) accessibility and user-friendliness of e-consent, 2) user engagement and comprehension, 3) customisability to participant preferences and demographics, 4) data security and 5) impact on research teams.

CONCLUSION

E-consenting approaches are generally well received by participants, with most studies reporting user-friendly interfaces and sufficient participant comprehension of consenting documentation.

IMPLICATIONS FOR PRACTICE

E-consent may facilitate remotely-conducted research by offering a feasible and robust alternative to face-to-face consenting approaches, however paper-based options should still be offered, based on participant preference. Customising e-consenting platforms may improve accessibility for individuals with specific needs, and increase engagement with study information. Research teams must offer prospective participants opportunities to discuss study information in real-time.

Resistance training and health in adults: an overview of systematic reviews

The objective of this overview of systematic reviews was to determine the benefits and harms of resistance training (RT) on health outcomes in adults aged 18 years or older, compared with not participating in RT. Four electronic databases were searched in February 2019 for systematic reviews published in the past 10 years. Eligibility criteria were determined a priori for population (community dwelling adults), intervention (exclusively RT), comparator (no RT or different doses of RT), and health outcomes (critical: mortality, physical functioning, health-related quality of life, and adverse events; important: cardiovascular disease, type 2 diabetes mellitus, mental health, brain health, cognitive function, cancer, fall-related injuries or falls, and bone health). We selected 1 review per outcome and we used the GRADE process to assess the strength of evidence. We screened 2089 records and 375 full-text articles independently, in duplicate. Eleven systematic reviews were included, representing 364 primary studies and 382 627 unique participants. RT was associated with a reduction in all-cause mortality and cardiovascular disease incidence, and an improvement in physical functioning. Effects on health-related quality of life or cognitive function were less certain. Adverse events were not consistently monitored or reported in RT studies, but serious adverse events were not common. Systematic reviews for the remaining important health outcomes could not be identified. Overall, RT training improved health outcomes in adults and the benefits outweighed the harms. (PROSPERO registration no.: CRD42019121641.)

Novelty This overview was required to inform whether there was new evidence to support changes to the recommended guidelines for resistance training.

Interest in meta-research in geriatric medicine: a survey of members of the European Geriatric Medicine Society

PURPOSE

There has been an exponential increase in meta-research, especially in the branch dealing with systematic reviews [SRs] and meta-analyses [MAs]. However, the knowledge regarding these topics in geriatric medicine is still poorly explored. We therefore undertook a survey of the current knowledge and needs in meta-research in geriatrics.

METHODS

A short survey (taking approximately 5 min to complete) was freely available on the European Geriatric Medicine Society (EuGMS) website and disseminated via social networks by the EuGMS and the authors of the survey. The questionnaire was available during the whole year of 2019. The questionnaire specifically addressed demographic information, previous research activities and the knowledge of the participants on meta-research in geriatric medicine.

RESULTS

The survey was completed by 291 participants from 36 different countries of about 20,000 EuGMS members (mostly with an age between 55 and 64 years; 51.5% female; mainly from Italy and Germany). Most respondents (65.6%) reported reading more than 20 articles in the past year, but few (36.4%) read more than 10 SRs/MAs. Participants reported that SRs and/or MAs are important in clinical practice: 83.8% giving a score of > 6/10, and 23.0% reporting 10. The large majority of the participants asked for more education in meta-research. In particular, there is need for educational courses for meta-research in geriatric medicine, online or in person, organized by the EuGMS.

CONCLUSION

Our survey shows that interest in SRs/MAs is high, and there is a demand for education on these topics in relation to geriatric medicine.

Nutrition in times of Covid-19, how to trust the deluge of scientific information

PURPOSE OF REVIEW

The Covid-19 pandemic has daunted the world with its enormous impact on healthcare, economic recession, and psychological distress. Nutrition is an integral part of every person life care, and should also be mandatorily integrated to patient care under the Covid-19 pandemic. It is crucial to understand how the Covid-19 does develop and which risk factors are associated with negative outcomes and death. Therefore, it is of utmost importance to have studies that respect the basic tenets of the scientific method in order to be trusted. The goal of this review is to discuss the deluge of scientific data and how it might influence clinical reasoning and practice.

RECENT FINDINGS

A large number of scientific manuscripts are daily published worldwide, and the Covid-19 makes no exception. Up to now, data on Covid-19 have come from countries initially affected by the disease and mostly pertain either epidemiological observations or opinion papers. Many of them do not fulfil the essential principles characterizing the adequate scientific method.

SUMMARY

It is crucial to be able to critical appraise the scientific literature, in order to provide adequate nutrition therapy to patients, and in particular, to Covid-19 infected individuals.

Assessment of Publication Trends of Systematic Reviews and Randomized Clinical Trials, 1995 to 2017

This cross-sectional study assesses the growth in the number of systematic reviews and meta-analyses vs randomized clinical trials from 1995 to 2017.

Systematic Review With Meta-Analysis of Patient-Centered Outcomes, Comparing International Guideline-Recommended Enteral Protein Delivery With Usual Care

BACKGROUND

International guidelines recommend that protein be administered enterally to critically ill patients at doses between 1.2 and 2 g/kg per day Observational data indicate that patients frequently receive less protein. The aim of this systematic review was to evaluate patient-centered outcomes with guideline-recommended enteral protein compared with usual care.

METHODS

A systematic review was performed of randomized controlled trials including critically ill adult patients provided predominately enteral nutrition with mean protein at ≥1.2 g/kg per day (intervention) and <1.2 g/kg per day (comparator). Random-effects models were applied for outcomes reported in ≥3 trials.

RESULTS

Of 1375 abstracts, 69 full-text articles were reviewed, and 6 trials meet the inclusion criteria, including 511 patients. The intervention group received a mean (SD) of 1.3 (0.08) g/kg per day, and the comparator group received 0.75 (0.15) g/kg per day protein. Insufficient data were available for meta-analyses on the primary outcome (muscle mass or strength). According to our meta-analyses, mortality at 28 days (5 studies) (risk ratio 0.92 [95% Cl 0.63-1.35], P = .66) and the durations of intensive care unit (6 studies) and hospital admission (4 studies) were similar between the intervention and comparator, with some uncertainty due to sample sizes and heterogeneity.

CONCLUSION

There are insufficient data to conclude whether protein provision within the current international guideline recommendations improves outcomes. In a limited dataset, enteral protein intakes near the lower level of current recommendations do not appear to reduce admission duration or mortality when compared with usual care in critically ill.

The commercialization of the biomedical sciences: (mis)understanding bias

The growing commercialization of scientific research has raised important concerns about industry bias. According to some evidence, so-called industry bias can affect the integrity of the science as well as the direction of the research agenda. I argue that conceptualizing industry's influence in scientific research in terms of bias is unhelpful. Insofar as industry sponsorship negatively affects the integrity of the research, it does so through biasing mechanisms that can affect any research independently of the source of funding. Talk about industry bias thus offers no insight into the particular epistemic shortcomings at stake. If the concern is with the negative effects that industry funding can have on the research agenda, conceptualizing this influence as bias obscures the ways in which such impact is problematic and limits our ability to offer solutions that can successfully address the concerns raised by the growing role of private funding in science.

A critical review of the role of milk and other dairy products in the development of obesity in children and adolescents

Existing reviews suggest that milk and other dairy products do not play a role in the development of obesity in childhood, but they do make an important contribution to children's nutrient intake. It is thus curious that public health advice on the consumption of dairy products for children is often perceived as unclear. The present review aimed to provide an overview of the totality of the evidence on the association between milk and other dairy products, and obesity and indicators of adiposity, in children. Our search identified forty-three cross-sectional studies, thirty-one longitudinal cohort studies and twenty randomised controlled trials. We found that milk and other dairy products are consistently found to be not associated, or inversely associated, with obesity and indicators of adiposity in children. Adjustment for energy intake tended to change inverse associations to neutral. Also, we found little evidence to suggest that the relationship varied by type of milk or dairy product, or age of the children, although there was a dearth of evidence for young children. Only nine of the ninety-four studies found a positive association between milk and other dairy products and body fatness. There may be some plausible mechanisms underlying the effect of milk and other dairy products on adiposity that influence energy and fat balance, possibly through fat absorption, appetite or metabolic activity of gut microbiota. In conclusion, there is little evidence to support a concern to limit the consumption of milk and other dairy products for children on the grounds that they may promote obesity.

Study-based registers reduce waste in systematic reviewing: discussion and case report

BACKGROUND

Maintained study-based registers (SBRs) have, at their core, study records linked to, potentially, multiple other records such as references, data sets, standard texts and full-text reports. Such registers can minimise and refine searching, de-duplicating, screening and acquisition of full texts. SBRs can facilitate new review titles/updates and, within seconds, inform the team about the potential workload of each task.

METHODS

We discuss the advantages/disadvantages of SBRs and report a case of how such a register was used to develop a successful grant application and deliver results-reducing considerable redundancy of effort.

RESULTS

SBRs saved time in question-setting and scoping and made rapid production of nine Cochrane systematic reviews possible.

CONCLUSION

Whilst helping prioritise and conduct systematic reviews, SBRs improve quality. Those funding information specialists for literature reviewing could reasonably stipulate the resulting SBR to be delivered for dissemination and use beyond the life of the project.

Characteristics, trend, and methodological quality of systematic reviews and meta-analyses in nuclear medicine: A bibliometric analysis of studies published between 2005 and 2016

To evaluate the characteristics, trend, and quality of systematic reviews and meta-analyses in nuclear medicine.We performed a PubMed search to identify systematic reviews and meta-analyses published between 2005 and 2016 in the field of nuclear medicine. The following data were extracted: journal name, impact factor, type of study, topics with cancer type, imaging modalities, authors (number, country, affiliation, presence of nuclear medicine specialists and statisticians, discordance between the first and corresponding authors), funding, methodological quality, methods used for quality assessment, and statistical methods.We included 185 nuclear medicine articles. Meta-analyses (n = 164; 88.6%) were published about 7 times more frequently than systematic reviews. Oncology was the most commonly studied topic (n = 125, 67.6%). The first authors were most frequently located in China (n = 73; 39.5%). PET was the most commonly used modality (n = 150; 81.1%). Both the number of authors and the ratio of discordance between the first and corresponding authors tended to progressively increase over time.The mean AMSTAR score increased over time (5.77 in 2005-2008, 6.71 in 2009-2012, and 7.44 in 2013-2016). The proportion of articles with quality assessment increased significantly (20/26 in 2005-2008, 54/65 in 2009-2012, and 79/94 in 2013-2016). The most commonly used assessment tool was quality assessment of diagnostic accuracy studies (n = 85; 54.9%).The number and quality of systematic reviews and meta-analyses in nuclear medicine have significantly increased over the review period; however, the quality of these articles varies. Efforts to overcome specific weaknesses of the methodologies can provide opportunities for quality improvement.

Bubble effect: including internet search engines in systematic reviews introduces selection bias and impedes scientific reproducibility

BACKGROUND

Using internet search engines (such as Google search) in systematic literature reviews is increasingly becoming a ubiquitous part of search methodology. In order to integrate the vast quantity of available knowledge, literature mostly focuses on systematic reviews, considered to be principal sources of scientific evidence at all practical levels. Any possible individual methodological flaws present in these systematic reviews have the potential to become systemic.

MAIN TEXT

This particular bias, that could be referred to as (re)search bubble effect, is introduced because of inherent, personalized nature of internet search engines that tailors results according to derived user preferences based on unreproducible criteria. In other words, internet search engines adjust their user's beliefs and attitudes, leading to the creation of a personalized (re)search bubble, including entries that have not been subjected to rigorous peer review process. The internet search engine algorithms are in a state of constant flux, producing differing results at any given moment, even if the query remains identical. There are many more subtle ways of introducing unwanted variations and synonyms of search queries that are used autonomously, detached from user insight and intent. Even the most well-known and respected systematic literature reviews do not seem immune to the negative implications of the search bubble effect, affecting reproducibility.

CONCLUSION

Although immensely useful and justified by the need for encompassing the entirety of knowledge, the practice of including internet search engines in systematic literature reviews is fundamentally irreconcilable with recent emphasis on scientific reproducibility and rigor, having a profound impact on the discussion of the limits of scientific epistemology. Scientific research that is not reproducible, may still be called science, but represents one that should be avoided. Our recommendation is to use internet search engines as an additional literature source, primarily in order to validate initial search strategies centered on bibliographic databases.