The methodological quality of systematic reviews on the treatment of adult major depression needs improvement according to AMSTAR 2: A cross-sectional study

Nutritional assessment in acute stroke patients: A systematic review of guidelines and systematic reviews

INTRODUCTION

Dysphagia and malnutrition are major contributors to mortality in patients with acute stroke. An early assessment of nutritional status upon hospital admission is crucial to enhance clinical outcomes by reducing the associated high-risk complications. However, the fragmented nature of the existing literature makes it difficult to optimize clinical practices.

OBJECTIVE

This study aims to identify the best clinical practices that nurses and other healthcare professionals can employ for the immediate assessment of nutritional risk in patients diagnosed with acute stroke.

DESIGN

Systematic review of clinical practice guidelines and systematic reviews.

DATA SOURCES

Comprehensive bibliographic searches were conducted up to May 2024 following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines across the databases Cochrane Library, PubMed, Embase, CINAHL, and Scopus, and three recognized guideline repositories.

REVIEW METHODS

The quality of clinical practice guidelines was ascertained using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) instrument, and the quality of systematic reviews was assessed through A MeaSurement Tool to Assess Systematic Reviews (AMSTAR 2). The evidence quality was appraised based on the classifications by the European Society of Cardiology.

RESULTS

Out of 2534 identified records, 15 were incorporated into this review. Predominantly, the selected clinical practice guidelines and systematic reviews exhibited high methodological quality. Notably, the Gugging Swallowing Screen and the Malnutrition Universal Screening Tool were pinpointed as primary tools for initial screenings. Most studies recommended that these assessments, ideally conducted within the first 24 h of patient admission, should be carried out by specially trained professionals, highlighting the pivotal role of nurses. Deviations in screening outcomes necessitate complementary specialist evaluations.

CONCLUSIONS

This systematic review offers a consolidation of current insights, proposing an innovative and integrated approach to assess nutritional needs of high-risk patients. It underscores the importance of nurses in the screening process, emphasizing their pivotal role in the nutritional management of patients with acute stroke, and advocates for further research endeavors to standardize intervention protocols to elevate patient clinical outcomes.

PROTOCOL REGISTRATION NUMBER

PROSPERO CRD42023425140.

Effectiveness of solution-focused brief therapy: An umbrella review of systematic reviews and meta-analyses

OBJECTIVE

This umbrella review of systematic reviews and meta-analyses evaluates the effectiveness of Solution-Focused Brief Therapy (SFBT) across various populations and settings.

METHOD

A total of 25 systematic reviews, including 15 meta-analyses, were analyzed. Reviews were included if they used systematic search methods and quality assessments and focused on the effectiveness of SFBT as a therapeutic approach.

RESULTS

SFBT demonstrated significant positive outcomes across different issues, settings, and cultural contexts, with no evidence of harm. High confidence in evidence of effectiveness was established for depression, overall mental health, and progress towards individual goals for the adult population. In addition, findings indicated mainly moderate confidence in evidence of SFBT effectiveness for a wide variety of outcomes for all age groups. No difference was found in the confidence in the evidence by world region, though Western and Eastern studies researched some different aspects.

CONCLUSION

SFBT is an effective therapeutic approach for various psychological, social, school, medical, couple, or self-related issues. Further research with rigorous methodologies and comprehensive reporting is needed to strengthen the confidence in these findings and provide evidence for the brevity of the intervention.

Characteristics and quality of systematic reviews led by Peruvian authors: A scoping review

Background

Systematic reviews (SRs) worldwide suffer from methodological deficiencies, potentially biasing intervention decisions, and Peruvian SRs are no exception. Evaluating SRs led by Peruvian researchers is a crucial step to enhance quality and transparency in decision-making and to identify topics where SRs are either scarce or prioritized for research.

Objective

To describe the characteristics and assess the methodological quality of SRs with Peruvian first authors.

Methods

We conducted a scoping review within the Scopus database on January 5, 2023. We aimed to identify published SRs of interventions in which the first author had a Peruvian affiliation, published between 2013 and 2022. We evaluated the methodological quality of these SRs using the AMSTAR 2 tool. We assessed the factors associated with the AMSTAR 2 score using adjusted mean differences (aMD), including their 95 % confidence intervals (95 % CI).

Results

We identified 95 eligible SRs, with a clear upward trend. SRs were primarily published in Q1 (43.2 %) and Q2 (23.2 %) journals, predominantly affiliated with institutions in Lima (90.5 %). Areas like infectious diseases (20.0 %) and dentistry (18.9 %) were most frequent. AMSTAR 2 assessments highlighted deficiencies, with few SRs reporting prior protocols (37.9 %), comprehensive search strategies (23.2 %), explanations for excluded studies (20.0 %), adequate descriptions of included studies (38.3 %), or funding sources (19.1 %). Notably, SRs in Q4 journals (aMD: -19.7, 95 % CI: -33.8 to -5.5) and those on surgical interventions (aMD: -22.6, 95 % CI: -34.7 to -10.4) had lower AMSTAR 2 scores.

Conclusions

Although Peruvian-led SRs are increasingly being published, critical deficiencies are common, especially in reporting protocols, search strategies, study descriptions, and funding sources. Addressing these gaps is pivotal for enhancing the credibility and utility of these SRs in informing decision-making.

Effectiveness of eHealth interventions for HIV prevention, testing and management: An umbrella review

BACKGROUND

Human immunodeficiency virus (HIV) infection has become a major contributor to the global burden of disease. Globally, the number of cases of HIV continues to increase. Electronic health (eHealth) interventions have emerged as promising tools to support disease self-management among people living with HIV. The purpose of this umbrella review is to systematically evaluate and summarize the evidence and results of published systematic reviews and meta-analyses on the effectiveness of eHealth interventions for HIV prevention, testing and management.

METHODS

PubMed, Embase and the Cochrane Library were searched for reviews. The methodological quality of the included studies was assessed using AMSTAR-2.

RESULTS

A total of 22 systematic reviews were included. The methodological quality of the reviews was low or critically low. EHealth interventions range from Internet, computer, or mobile interventions to websites, programs, applications, email, video, games, telemedicine, texting, and social media, or a combination of them. The majority of the reviews showed evidence of effectiveness (including increased participation in HIV management behaviours, successfully changed HIV testing behaviours, and reduced risk behaviours). EHealth interventions were effective in the short term.

CONCLUSIONS

Ehealth interventions have the potential to improve HIV prevention, HIV testing and disease management. Due to the limitations of the low methodological quality of the currently available systematic reviews, more high-quality evidence is needed to develop clear and robust recommendations.

Analysis of risk of bias assessments in a sample of intervention systematic reviews, part I: many aspects of conduct and reporting need improvement

OBJECTIVES

Current standards for systematic reviews (SRs) require adequate conduct and complete reporting of risk of bias (RoB) assessments of the individual studies included in the review. We investigated the conduct and reporting of RoB assessments reported in a sample of SRs of interventions for persons with cerebral palsy (CP).

STUDY DESIGN AND SETTING

We included SRs published from 2014 to 2021. Authors worked in pairs to independently extract data on the characteristics of the SRs and to rate their conduct and reporting. The conduct of RoB assessment was appraised with the three AMSTAR-2 items related to RoB assessment. Reporting completeness was evaluated using the two items related to RoB assessment within studies in the PRISMA 2020 guidelines. We use descriptive statistics to report the consensus data, in accordance with our protocol.

RESULTS

We included 145 SRs. Among the 128 (88.3%) SRs that assessed RoB, the standards for AMSTAR-2 item 9 (use of an adequate RoB tool) were partially or fully satisfied in 73 (57.0%). Across the 128 SRs that assessed RoB, 46 (35.9%) accounted for RoB in interpreting the SR's findings and, of the 49 that included a meta-analysis, 11 (22.4%) discussed the impact of RoB on this. 123 (96.1%) of the 128 SRs named the RoB tool that was used for at least one of the study designs they included, 96 (75.0%) specified the RoB items assessed and 89 (69.5%) reported the findings for each item, 81 (63.2%) fully reported the processes for RoB assessment, 68 (53.1%) reported how an overall RoB judgment was reached, and 74 (57.8%) reported an overall RoB assessment for every study.

CONCLUSION

The selection and application of RoB tools in this sample of SRs about interventions for CP are comparable to those reported in other recent studies. However, most SRs in this sample did not fully meet the appraisal standards of AMSTAR-2 regarding the adequacy of the RoB tool applied and other aspects of RoB assessment conduct; Cochrane SRs were a notable exception. Overall, reporting of RoB assessments was somewhat better than conduct, perhaps reflecting the more widespread uptake of the PRISMA guidelines. Our findings may be generalizable to some extent, considering the extensive literature reporting widespread inadequacies in health care-related intervention SRs and reports from other specialties that document similar RoB assessment deficiencies. As such, this study should remind authors, peer reviewers, and journal editors to follow the RoB assessment reporting guidelines of PRISMA 2020 and to understand the corresponding critical appraisal standards of AMSTAR-2. We recommend a shift of focus from the documentation of inadequate RoB assessments and well-known deficiencies in other components of SRs towards the implementation of changes to address these problems along with plans to evaluate their effectiveness.

Pediatric dentistry systematic reviews using the GRADE approach: methodological study

BACKGROUND

To assess the reporting of the certainty of the evidence using the GRADE approach in systematic reviews of interventions in pediatric dentistry.

METHODS

The inclusion criteria were systematic reviews of randomized clinical trials (RCTs) and non-randomized studies of interventions (NRSIs) in pediatric dentistry that reported the certainty of the evidence through the GRADE approach. Paired independent reviewers screened the studies, extracted data, and appraised the methodological quality using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR 2) tool. The certainty of the evidence was extracted for each outcome. A descriptive analysis was conducted.

RESULTS

Around 28% of pediatric dentistry reviews of interventions used the GRADE approach (n = 24). Twenty reviews reported 112 evidence outcomes from RCTs and 13 from NRSIs using GRADE evidence profile tables. The methodological quality was high (16.7%), moderate (12.5%), low (37.5%), and critically low (33.3%), fulfilling the majority of the AMSTAR 2 criteria. The certainty of the evidence for outcomes generated from RCTs and NRSIs was very low (40.2% and 84.6%), low (33.1% and 7.7%), moderate (17.8% and 7.7%), and high (9.8% and 0.0%). The main reasons to downgrade the certainty were due to (for RCTs and NRSIs, respectively): risk of bias (68.8% and 84.6%), imprecision (67.8% and 100.0%), inconsistency (18.8% and 23.1%), indirectness (17.8% and 0.0%), and publication bias (7.1% and 0.0%).

CONCLUSION

The proportion of systematic reviews assessing the certainty of the evidence using the GRADE approach was considered small, considering the total initial number of published pediatric dentistry reviews of intervention. The certainty of the evidence was mainly very low and low, and the main problems for downgrading the certainty of evidence were due to risk of bias and imprecision.

REGISTRATION

PROSPERO database #CRD42022365443.

Deprescribing interventions in older adults: An overview of systematic reviews

OBJECTIVE

The growing deprescribing field is challenged by a lack of consensus around evidence and knowledge gaps. The objective of this overview of systematic reviews was to summarize the review evidence for deprescribing interventions in older adults.

METHODS

11 databases were searched from 1st January 2005 to 16th March 2023 to identify systematic reviews. We summarized and synthesized the results in two steps. Step 1 summarized results reported by the included reviews (including meta-analyses). Step 2 involved a narrative synthesis of review results by outcome. Outcomes included medication-related outcomes (e.g., medication reduction, medication appropriateness) or twelve other outcomes (e.g., mortality, adverse events). We summarized outcomes according to subgroups (patient characteristics, intervention type and setting) when direct comparisons were available within the reviews. The quality of included reviews was assessed using A MeaSurement Tool to Assess systematic Reviews 2 (AMSTAR 2).

RESULTS

We retrieved 3,228 unique citations and assessed 135 full-text articles for eligibility. Forty-eight reviews (encompassing 17 meta-analyses) were included. Thirty-one of the 48 reviews had a general deprescribing focus, 16 focused on specific medication classes or therapeutic categories and one included both. Twelve of 17 reviews meta-analyzed medication-related outcomes (33 outcomes: 25 favored the intervention, 7 found no difference, 1 favored the comparison). The narrative synthesis indicated that most interventions resulted in some evidence of medication reduction while for other outcomes we found primarily no evidence of an effect. Results were mixed for adverse events and few reviews reported adverse drug withdrawal events. Limited information was available for people with dementia, frailty and multimorbidity. All but one review scored low or critically low on quality assessment.

CONCLUSION

Deprescribing interventions likely resulted in medication reduction but evidence on other outcomes, in particular relating to adverse events, or in vulnerable subgroups or settings was limited. Future research should focus on designing studies powered to examine harms, patient-reported outcomes, and effects on vulnerable subgroups.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42020178860.

Systematic review of clinical practice guidelines and systematic reviews: A method for conducting comprehensive analysis

A systematic review (SR) is a research method for synthesizing evidence on a specific topic. Among the various types of systematic reviews, there are SRs of guidelines (CPGs) and SRs of SRs. Traditionally, they are limited to just one type of secondary evidence. This paper introduces an innovative SR methodology that combines CPGs and SRs to improve evidence synthesis and overcome the limitations of isolated use. Essential steps that should always precede the actual research process include registering the research protocol, formulating research questions and setting inclusion/exclusion criteria. Using the PRISMA protocol for comprehensive database searches, it's crucial to combine keywords with boolean operators and remove duplicates. The eligibility of studies should be assessed by selecting potentially relevant articles through an initial screening of titles and abstracts, followed by a meticulous analysis of the full-texts. Rigorous evidence evaluation tools, such as AGREE II for CPGs and AMSTAR 2 for SRs, and the double reviewer approach ensure high-quality selections. Additionally, converting summarized results into percentages and applying statistical analyses facilitate interpretation and improve the reliability of rater assessments. A further characteristic of this methodology is its adaptability to the evolution of healthcare research.

The confidence in the results of physiotherapy systematic reviews in the musculoskeletal field is not increasing over time: a meta-epidemiological study using AMSTAR 2 tool

OBJECTIVES

To assess the confidence in the results of systematic reviews on the effectiveness of physiotherapy for musculoskeletal conditions in the past 10 years and to analyze trends and factors associated.

METHODS

This is a metaepidemiological study on systematic reviews (SRs) with meta-analysis of randomized controlled trials (RCTs). MEDLINE, Cochrane Database of Systematic Reviews, CINAHL, and PEDro were searched for SRs of RCT on physiotherapy interventions for musculoskeletal disorders from December 2012 to December 2022. Two researchers independently screened the records based on the inclusion criteria; a random sample of 100 studies was selected, and each journal, author, and study variable was extracted. The methodological quality of SRs was independently assessed with the AMSTAR 2 tool. Any disagreement was solved by consensus.

RESULTS

The confidence in SRs results was critically low in 90% of the studies, and it did not increase over time. Cochrane reviews are predominantly represented in the higher AMSTAR 2 confidence levels, with a statistically significant difference compared to non-Cochrane reviews. The last author's H-index is the only predictor of higher confidence among the variables analyzed (OR 1.04; 95% CI: 1.01, 1.06).

CONCLUSION

The confidence in SRs results is unacceptably low. Given the relevance of musculoskeletal disorders and the impact of evidence synthesis on the clinical decision-making process, there is an urgent need to improve the quality of secondary research by adopting more rigorous methods.

Overview of systematic reviews on periodontal-orthodontic interactions: A comprehensive literature analysis

The aims of this research were to investigate the methodological quality of systematic reviews on periodontal-orthodontic interactions (i.e. reviews of primary research broadly defined as any including both periodontic and orthodontic components) and to provide a mapping of the researched topics. We searched four major databases (PubMed, Lilacs, Web of Science, and Embase) for systematic reviews of periodontal-orthodontic interactions. We used the AMSTAR-2 tool (the acronym is derived from 'a measurement tool to assess systematic reviews') to assess the methodological quality of the included systematic reviews. Individual AMSTAR-2 ratings were tabulated, and the percentage per item was calculated. To assess the association between the AMSTAR-2 percentage score and the overall confidence in the systematic review results, an ordinal regression model was used. We initially retrieved 973 documents, and 43 systematic reviews were included. Systematic reviews of interventions were the most prevalent (n = 26, 60.5%). Most of the systematic reviews did not report a meta-analysis (n = 25, 58.1%). In addition, most of the studies included in the systematic reviews had an unclear or high risk of bias. Most of the systematic reviews were rated as having critically low or low overall confidence (n = 34, 79.1%). A significant correlation was found between the AMSTAR-2 percentage score and overall confidence in the results. The methodological quality of systematic reviews on periodontal-orthodontic interactions can be improved. The limitations of our study include potential language bias and an arbitrary classification of the topics researched.

Appraisal methods and outcomes of AMSTAR 2 assessments in overviews of systematic reviews of interventions in the cardiovascular field: A methodological study

This study aimed to assess the methods and outcomes of The Measurement Tool to Assess systematic Reviews (AMSTAR) 2 appraisals in overviews of reviews (overviews) of interventions in the cardiovascular field and identify factors that are associated with these outcomes. MEDLINE, Scopus, and the Cochrane Database of Systematic Reviews were searched until November 2022. Eligible were overviews of cardiovascular interventions, analyzing systematic reviews (SRs) of randomized controlled trials (RCTs). Extracted data included characteristics of overviews and SRs and AMSTAR 2 appraisal methods and outcomes. Data were synthesized using descriptive statistics and logistic regression to explore potential associations between the characteristics of SRs and extracted AMSTAR 2 overall ratings ("High-Moderate" vs. "Low-Critically low"). The original results on individual AMSTAR 2 items were entered into the official AMSTAR 2 online tool and the recalculated overall confidence ratings were compared to those provided in overviews. All 34 overviews identified were published between 2019 and 2022. Rating of overall confidence following the algorithm suggested by AMSTAR 2 developers was noted in 74% of overviews. The 679 unique included SRs were mainly of "Critically low" (53%) or "Low" (18.7%) confidence and underperformed in items 2 (Protocol, no = 65.2%) and 7 (List of excluded studies, no = 84%). The following characteristics of SRs were significantly associated with higher overall ratings: Cochrane origin, pharmacological interventions, including exclusively RCTs, citation of methodological and reporting guidelines, protocol, absence of funding and publication after AMSTAR 2 release. Generally, overviews' authors tended to deviate from the original rating scheme and ascribe higher ratings to SRs compared to the official AMSTAR 2 online tool. Most SRs included in overviews of cardiovascular interventions have critically low or low confidence in their results. Overviews' authors should be more transparent about the methods used to derive the overall confidence in SRs.

Efficacy and safety of ketamine and esketamine for unipolar and bipolar depression: an overview of systematic reviews with meta-analysis

Background

Unipolar and bipolar depression present treatment challenges, with patients sometimes showing limited or no response to standard medications. Ketamine and its enantiomer, esketamine, offer promising alternative treatments that can quickly relieve suicidal thoughts. This Overview of Reviews (OoR) analyzed and synthesized systematic reviews (SRs) with meta-analysis on randomized clinical trials (RCTs) involving ketamine in various formulations (intravenous, intramuscular, intranasal, subcutaneous) for patients with unipolar or bipolar depression. We evaluated the efficacy and safety of ketamine and esketamine in treating major depressive episodes across various forms, including unipolar, bipolar, treatment-resistant, and non-resistant depression, in patient populations with and without suicidal ideation, aiming to comprehensively assess their therapeutic potential and safety profile.

Methods

Following PRIOR guidelines, this OoR's protocol was registered on Implasy (ID:202150049). Searches in PubMed, Scopus, Cochrane Library, and Epistemonikos focused on English-language meta-analyses of RCTs of ketamine or esketamine, as monotherapy or add-on, evaluating outcomes like suicide risk, depressive symptoms, relapse, response rates, and side effects. We included studies involving both suicidal and non-suicidal patients; all routes and formulations of administration (intravenous, intramuscular, intranasal) were considered, as well as all available comparisons with control interventions. We excluded meta-analysis in which the intervention was used as anesthesia for electroconvulsive therapy or with a randomized ascending dose design. The selection, data extraction, and quality assessment of studies were carried out by pairs of reviewers in a blinded manner. Data on efficacy, acceptability, and tolerability were extracted.

Results

Our analysis included 26 SRs and 44 RCTs, with 3,316 subjects. The intervention is effective and well-tolerated, although the quality of the included SRs and original studies is poor, resulting in low certainty of evidence.

Limitations

This study is limited by poor-quality SRs and original studies, resulting in low certainty of the evidence. Additionally, insufficient available data prevents differentiation between the effects of ketamine and esketamine in unipolar and bipolar depression.

Conclusion

While ketamine and esketamine show promising therapeutic potential, the current evidence suffers from low study quality. Enhanced methodological rigor in future research will allow for a more informed application of these interventions within the treatment guidelines for unipolar and bipolar depression.

Systematic review registration

[https://inplasy.com/inplasy-2021-5-0049/], identifier (INPLASY202150049).

Investigate the relationship between the retraction reasons and the quality of methodology in non-Cochrane retracted systematic reviews: a systematic review

BACKGROUND

This systematic review aimed to investigate the relationship between retraction status and the methodology quality in the retracted non-Cochrane systematic review.

METHOD

PubMed, Web of Science, and Scopus databases were searched with keywords including systematic review, meta-analysis, and retraction or retracted as a type of publication until September 2023. There were no time or language restrictions. Non-Cochrane medical systematic review studies that were retracted were included in the present study. The data related to the retraction status of the articles were extracted from the retraction notice and Retraction Watch, and the quality of the methodology was evaluated with the AMSTAR-2 checklist by two independent researchers. Data were analyzed in the Excel 2019 and SPSS 21 software.

RESULT

Of the 282 systematic reviews, the corresponding authors of 208 (73.75%) articles were from China. The average interval between publish and retraction of the article was about 23 months and about half of the non-Cochrane systematic reviews were retracted in the last 4 years. The most common reasons for retractions were fake peer reviews and unreliable data, respectively. Editors and publishers were the most retractors or requestors for retractions. More than 86% of the retracted non-Cochrane SRs were published in journals with an impact factor above two and had a critically low quality. Items 7, 9, and 13 among the critical items of the AMSTAR-2 checklist received the lowest scores.

DISCUSSION AND CONCLUSION

There was a significant relationship between the reasons of retraction and the quality of the methodology (P-value < 0.05). Plagiarism software and using the Cope guidelines may decrease the time of retraction. In some countries, strict rules for promoting researchers increase the risk of misconduct. To avoid scientific errors and improve the quality of systematic reviews/meta-analyses (SRs/MAs), it is better to create protocol registration and retraction guidelines in each journal for SRs/MAs.

Guidance to best tools and practices for systematic reviews

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Most systematic reviews reporting adherence to AMSTAR 2 had critically low methodological quality: a cross-sectional meta-research study

OBJECTIVES

To analyze the methodological quality and characteristics of systematic reviews (SRs) that reported they were conducted in line with the AMSTAR 2 (A MeaSurement Tool to Assess Systematic Reviews).

STUDY DESIGN AND SETTING

This was a cross-sectional meta-research study. We searched MEDLINE and Embase. We included full reports of SRs reporting the study was conducted, prepared, or designed in line with the AMSTAR 2. Eligible SRs were those published from January 1, 2018, until May 3, 2022. We assessed the methodological quality of the included SRs using AMSTAR 2.

RESULTS

We included a total of 45 records. There were 43 SRs and 2 SR protocols. Among them, most were SRs of interventions that included primary studies on humans. More than half had a meta-analysis. According to our overall AMSTAR 2 assessments of included SRs, 35 SRs were of critically low confidence, 7 SRs were of low confidence, and one SR was of high confidence. There were no SRs of moderate confidence.

CONCLUSION

Even when authors indicate in their manuscripts that the SR was conducted/prepared/designed in line with the AMSTAR 2, it does not necessarily imply it is of high or even moderate confidence according to AMSTAR 2. A self-assessment with AMSTAR 2 could be required for submission and carefully checked by the editors/peer reviewers.

Methodological assessment and overall confidence in the results of systematic reviews with meta-analyses focusing on traumatic dental injuries: A cross-sectional study

BACKGROUND/AIMS

High methodological quality is required to interpret results of systematic reviews (SRs) in a reliable and accurate manner. The primary aim of this study was to appraise the methodologic quality of SRs with meta-analysis within the field of traumatic dental injuries using the A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 2 tool and assess overall confidence in their results. A secondary aim was to identify potential predictive factors associated with methodological quality.

MATERIALS AND METHODS

SRs with meta-analyses published in English in the field of traumatic dental injuries from inception to March 2023 were identified. The methodological quality of the included reviews was assessed using the AMSTAR 2 checklist. Two independent evaluators scored each AMSTAR 2 item as "yes" if it was adequately addressed, "partial yes" if it was partially addressed, and "no" if it was not addressed. The overall confidence in the results of each review was classified as "High," "Moderate," "Low," or "Critically low." Using multiple regression, the relationship between five predictor variables (journal impact factor, year of publication, number of authors, journal adherence to Preferred Reporting Items for Systematic reviews and Meta-analyses [PRISMA] guidelines and a priori protocol registration) and the total AMSTAR 2 scores was analyzed. The p-value was 5%.

RESULTS

Forty-one SRs were included. The overall confidence in the results of 13 reviews was categorized as "Critically low," 18 as "Low," 3 as "Moderate" and 7 as "High." Among the five predictor variables analyzed statistically, impact factor of the journal and year of publication significantly influenced the total AMSTAR 2 scores. The number of authors, adherence to PRISMA guidelines, and a priori protocol registration had no significant impact on AMSTAR 2 scores.

CONCLUSION

The overall confidence in the results of SRs with meta-analysis within the field of traumatic dental injuries was "Low" or "Critically Low" in the vast majority of studies (31 of 41). SRs with meta-analyses published in journals with higher impact factors and more recent publications had significantly higher methodological quality.

Guidance to best tools and practices for systematic reviews

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Methodological and reporting quality assessment of network meta-analyses in anesthesiology: a systematic review and meta-epidemiological study

PURPOSE

The scientific rigour of the conduct and reporting of anesthesiology network meta-analyses (NMAs) is unknown. This systematic review and meta-epidemiological study assessed the methodological and reporting quality of NMAs in anesthesiology.

METHODS

We searched four databases, including MEDLINE, PubMed, Embase, and the Cochrane Systematic Reviews Database, for anesthesiology NMAs published from inception to October 2020. We assessed the compliance of NMAs against A Measurement Tool to Assess Systematic Reviews (AMSTAR-2), Preferred Reporting Items for Systematic Reviews and Meta-Analyses Statement for Network Meta-Analyses (PRISMA-NMA), and PRISMA checklists. We measured the compliance across various items in AMSTAR-2 and PRISMA checklists and provided recommendations to improve quality.

RESULTS

Using the AMSTAR-2 rating method, 84% (52/62) of NMAs were rated "critically low." Quantitatively, the median [interquartile range] AMSTAR-2 score was 55 [44-69]%, while the PRISMA score was 70 [61-81]%. Methodological and reporting scores showed a strong correlation (R = 0.78). Anesthesiology NMAs had a higher AMSTAR-2 score and PRISMA score if they were published in higher impact factor journals (P = 0.006 and P = 0.01, respectively) or followed PRISMA-NMA reporting guidelines (P = 0.001 and P = 0.002, respectively). Network meta-analyses from China had lower scores (P < 0.001 and P < 0.001, respectively). Neither score improved over time (P = 0.69 and P = 0.67, respectively).

CONCLUSION

The current study highlights numerous methodological and reporting deficiencies in anesthesiology NMAs. Although the AMSTAR tool has been used to assess the methodological quality of NMAs, dedicated tools for conducting and assessing the methodological quality of NMAs are urgently required.

STUDY REGISTRATION

PROSPERO (CRD42021227997); first submitted 23 January 2021.

Guidance to best tools and practices for systematic reviews

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Guidance to best tools and practices for systematic reviews

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Guidance to best tools and practices for systematic reviews

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Guidance to Best Tools and Practices for Systematic Reviews

» Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.» A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.» Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Quality assessment of systematic reviews with meta-analysis in undergraduate nursing education

BACKGROUND

Little attention has been given to the methodological quality of meta-analyses in nursing education. This warrants further improvements in meta-analyses in nursing education.

OBJECTIVE

This study aimed to assess the methodological quality of meta-analysis in the field of undergraduate nursing education.

DESIGN

This was a methodological study to review the methodological quality of systematic reviews (SRs) with meta-analysis.

METHODS

Exhaustive literature searches were performed using five comprehensive databases. Between 1994 and 2022, 11,827 studies were identified, and 41 full-text articles met the inclusion criteria. Two researchers extracted data using A Measurement Tool to Assess Systematic Reviews (AMSTAR)-2. The Chi-square test was conducted to make comparisons before and after 2017, the year AMSTAR-2 was released.

RESULTS

A comprehensive literature retrieval strategy, inclusion and exclusion criteria, literature selection, and data extraction were observed in nursing education more than in other disciplines. Improvements to be made include pre-specifying the protocol, providing a list of excluded studies with their exclusion reasons, reporting the source of funding for the included studies, assessing and discussing the potential impact of risk of bias, as well as investigating and discussing publication bias and its impact.

CONCLUSIONS

The number of SRs with meta-analyses in nursing education is increasing. This warrants efforts to improve the quality of research. In addition, guidelines for reporting SRs in the field of nursing education should be constantly updated.

A Cross-Sectional Study Based on Forty Systematic Reviews of Foods with Function Claims (FFC) in Japan: Quality Assessment Using AMSTAR 2

BACKGROUND

The Foods with Function Claims (FFC) was introduced in Japan in April 2015 to make more products available that are labeled with health functions. The products' functionality of function claims must be explained by scientific evidence presented in systematic reviews (SRs), but the quality of recent SRs is unclear. This study assessed the quality of SRs in the FFC registered on the Consumer Affairs Agency (CAA) website in Japan.

METHODS

We searched the database from 1 April to 31 October 2022. Confidence in the methodological quality of each SR was evaluated by the AMSTAR 2 checklist.

RESULTS

Forty SRs were randomly extracted on the basis of the eligibility criteria and recruitment procedures. Overall confidence was rated as "high" (N = 0, 0%), "moderate" (N = 0, 0%), "low" (N = 2, 5%), or "critically low" (N = 38, 95%). The mean AMSTAR 2 score was 51.1% (SD 12.1%; range 19-73%). Among the 40 SRs, the number of critical domain deficiencies was 4 in 7.5% of SRs, 3 in 52.5% of SRs, 2 in 35% of SRs, and 1 in 5% of SRs. Registering the review's protocol and comprehensive search strategies were particularly common deficiencies. Additionally, the risk of bias (RoB) was insufficiently considered.

CONCLUSION

Overall, the methodological quality of the SRs based on the FFC, introduced eight years earlier, was very poor. This was especially true in the interpretation and discussion of critical domains, which had many deficiencies in terms of protocol registration, a comprehensive literature search strategy, and accounting for the RoB.

Definition, harms, and prevention of redundant systematic reviews

BACKGROUND

Along with other types of research, it has been stated that the extent of redundancy in systematic reviews has reached epidemic proportions. However, it was also emphasized that not all duplication is bad, that replication in research is essential, and that it can help discover unfortunate behaviors of scientists. Thus, the question is how to define a redundant systematic review, the harmful consequences of such reviews, and what we could do to prevent the unnecessary amount of this redundancy.

MAIN BODY

There is no consensus definition of a redundant systematic review. Also, it needs to be defined what amount of overlap between systematic reviews is acceptable and not considered a redundancy. One needs to be aware that it is possible that the authors did not intend to create a redundant systematic review. A new review on an existing topic, which is not an update, is likely justified only when it can be shown that the previous review was inadequate, for example, due to suboptimal methodology. Redundant meta-analyses could have scientific, ethical, and economic questions for researchers and publishers, and thus, they should be avoided, if possible. Potential solutions for preventing redundant reviews include the following: (1) mandatory prospective registration of systematic reviews; (2) editors and peer reviewers rejecting duplicate/redundant and inadequate reviews; (3) modifying the reporting checklists for systematic reviews; (4) developing methods for evidence-based research (EBR) monitoring; (5) defining systematic reviews; (6) defining the conclusiveness of systematic reviews; (7) exploring interventions for the adoption of methodological advances; (8) killing off zombie reviews (i.e., abandoned registered reviews); (9) better prevention of duplicate reviews at the point of registration; (10) developing living systematic reviews; and (11) education of researchers.

CONCLUSIONS

Disproportionate redundancy of the same or very similar systematic reviews can lead to scientific, ethical, economic, and societal harms. While it is not realistic to expect that the creation of redundant systematic reviews can be completely prevented, some preventive measures could be tested and implemented to try to reduce the problem. Further methodological research and development in this field will be welcome.

User experience of applying AMSTAR 2 to appraise systematic reviews of healthcare interventions: a commentary

BACKGROUND

'A Measurement Tool to Assess Systematic Reviews, version 2' (AMSTAR 2) is a validated 16-item scale designed to appraise systematic reviews (SRs) of healthcare interventions and to rate the overall confidence in their results. This commentary aims to describe the challenges with rating of the individual items and the application of AMSTAR 2 from the user perspective.

DISCUSSION

A group of six experienced users (methodologists working in different clinical fields for at least 10 years) identified and discussed the challenges in rating of each item and the general use of AMSTAR 2 to appraise SRs. A group discussion was used to develop recommendations on how users could deal with the identified challenges. We identified various challenges with the content of items 2-16 and with the derivation of the overall confidence ratings on AMSTAR 2. These challenges include the need (1) to provide additional definitions (e.g., what constitutes major deviations from SR protocol on item 2), (2) to choose a rating strategy for multiple conditions on single items (e.g., how to rate item 5 if studies were selected in duplicate, but consensus between two authors was not reported), and (3) to determine rules for deriving the confidence ratings (e.g., what items are critical for such ratings). Based on these challenges we formulated specific recommendations for items 2-16 that AMSTAR 2 users could consider before applying the tool. Our commentary adds to the existing literature by providing the first in-depth examination of the AMSTAR 2 tool from the user perspective. The identified challenges could be addressed by additional decision rules including definitions for ambiguous items and guidance for rating of complex items and derivation of confidence ratings. We recommend that a team consensus regarding such decision rules is required before appraisal procedure begins.

TRIAL REGISTRATION

Not applicable.

Different Approaches to Appraising Systematic Reviews of Digital Interventions for Physical Activity Promotion Using AMSTAR 2 Tool: Cross-Sectional Study

High-quality systematic reviews (SRs) can strengthen the evidence base for prevention and health promotion. A 16-item AMSTAR 2 tool allows the appraisal of SRs by deriving a confidence rating in their results. In this cross-sectional study, we aimed to assess and compare two approaches to appraising 30 SRs of digital interventions for physical activity (PA) promotion using AMSTAR 2. Approach 1 (appraisals with 2/16 items) was used to identify SRs with critically low confidence ratings. Approach 2 (appraisals with all 16 items) was used (1) to derive the confidence ratings, (2) to identify SR strengths and weaknesses, and (3) to compare SR strengths among subgroups of SRs. The appraisal outcomes were summarized and compared using descriptive statistics. Approach 1 was quick (mean of 5 min/SR) at identifying SRs with critically low confidence ratings. Approach 2 was slower (mean of 20 min/SR), but allowed to identify SR strengths and weaknesses. Approach 2 showed that confidence ratings were low to critically low in 29/30 SRs. More strengths were identified in SRs with review protocols relative to SRs without review protocols and in newer SRs (published after AMSTAR 2 release) relative to older SRs. Only two items on AMSTAR 2 can quickly identify SRs with critical weaknesses. Although most SRs received low to critically low confidence ratings, SRs with review protocols and newer SRs tended to have more strengths. Future SRs require review protocols and better adherence to reporting guidelines to improve the confidence in their results.

Benefits and Risks of Antidepressant Drugs During Pregnancy: A Systematic Review of Meta-analyses

BACKGROUND

The prescription of antidepressant drugs during pregnancy has been steadily increasing for several decades. Meta-analyses (MAs), which increase the statistical power and precision of results, have gained interest for assessing the safety of antidepressant drugs during pregnancy.

OBJECTIVE

We aimed to provide a meta-review of MAs assessing the benefits and risks of antidepressant drug use during pregnancy.

METHODS

Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, a literature search on PubMed and Web of Science databases was conducted on 25 October, 2021, on MAs assessing the association between antidepressant drug use during pregnancy and health outcomes for the pregnant women, embryo, fetus, newborn, and developing child. Study selection and data extraction were carried out independently and in duplicate by two authors. The methodological quality of included studies was evaluated with the AMSTAR-2 tool. Overlap among MAs was assessed by calculating the corrected covered area. Data were presented in a narrative synthesis, using four levels of evidence.

RESULTS

Fifty-one MAs were included, all but one assessing risks. These provided evidence for a significant increase in the risks for major congenital malformations (selective serotonin reuptake inhibitors, paroxetine, fluoxetine, no evidence for sertraline; eight MAs), congenital heart defects (paroxetine, fluoxetine, sertraline; 11 MAs), preterm birth (eight MAs), neonatal adaptation symptoms (eight MAs), and persistent pulmonary hypertension of the newborn (three MAs). There was limited evidence (only one MA for each outcome) for a significant increase in the risks for postpartum hemorrhage, and with a high risk of bias, for stillbirth, impaired motor development, and intellectual disability. There was inconclusive evidence, i.e., discrepant results, for an increase in the risks for spontaneous abortion, small for gestational age and low birthweight, respiratory distress, convulsions, feeding problems, and for a subsequent risk for autism with an early antidepressant drug exposure. Finally, MAs provided no evidence for an increase in the risks for gestational hypertension, preeclampsia, and for a subsequent risk for attention-deficit/hyperactivity disorder. Only one MA assessed benefits, providing limited evidence for preventing relapse in severe or recurrent depression. Effect sizes were small, except for neonatal symptoms (small to large). Results were based on MAs in which overall methodological quality was low (AMSTAR-2 score = 54.8% ± 12.9%, [19-81%]), with a high risk of bias, notably indication bias. The corrected covered area was 3.27%, which corresponds to a slight overlap.

CONCLUSIONS

This meta-review has implications for clinical practice and future research. First, these results suggest that antidepressant drugs should be used as a second-line treatment during pregnancy (after first-line psychotherapy, according to the guidelines). The risk of major congenital malformations could be prevented by observing guidelines that discourage the use of paroxetine and fluoxetine. Second, to decrease heterogeneity and bias, future MAs should adjust for maternal psychiatric disorders and antidepressant drug dosage, and perform analyses by timing of exposure.

Guidance to best tools and practices for systematic reviews1

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Effects of levels of automation and non-driving related tasks on driver performance and workload: A review of literature and meta-analysis

This study assessed the effects of different levels of automation and non-driving related tasks (NDRT) on driver performance and workload. A systematic literature review was conducted in March 2021 using Compendex, Google Scholar, Web of Science, and Scopus databases. Forty-five studies met the inclusion criteria. A meta-analysis was conducted and Cochrane risk of bias tool and Cochran's Q test were used to assess risk of bias and homogeneity of the effect sizes respectively. Results suggested that drivers exhibited safer performance when dealing with critical incidents in manual driving than partially automated driving (PAD) and highly automated driving (HAD) conditions. However, drivers reported higher workload in the manual driving mode as compared to the HAD and PAD conditions. Haptic, auditory, and visual-auditory takeover request modalities are preferred over the visual-only modality to improve takeover time. Use of handheld NDRTs significantly degraded driver performance as compared to NDRTs performed on mounted devices.

Can Communication Technologies Reduce Loneliness and Social Isolation in Older People? A Scoping Review of Reviews

BACKGROUND

Loneliness and social isolation in older age are considered major public health concerns and research on technology-based solutions is growing rapidly. This scoping review of reviews aims to summarize the communication technologies (CTs) (review question RQ1), theoretical frameworks (RQ2), study designs (RQ3), and positive effects of technology use (RQ4) present in the research field.

METHODS

A comprehensive multi-disciplinary, multi-database literature search was conducted. Identified reviews were analyzed according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) framework. A total of N = 28 research reviews that cover 248 primary studies spanning 50 years were included.

RESULTS

The majority of the included reviews addressed general internet and computer use (82% each) (RQ1). Of the 28 reviews, only one (4%) worked with a theoretical framework (RQ2) and 26 (93%) covered primary studies with quantitative-experimental designs (RQ3). The positive effects of technology use were shown in 55% of the outcome measures for loneliness and 44% of the outcome measures for social isolation (RQ4).

CONCLUSION

While research reviews show that CTs can reduce loneliness and social isolation in older people, causal evidence is limited and insights on innovative technologies such as augmented reality systems are scarce.

AMSTAR 2 appraisal of systematic reviews and meta-analyses in the field of heart failure from high-impact journals

BACKGROUND

The Measurement Tool to Assess systematic Reviews (AMSTAR) 2 is a critical appraisal tool for systematic reviews (SRs) and meta-analyses (MAs) of interventions. We aimed to perform the first AMSTAR 2-based quality assessment of heart failure-related studies.

METHODS

Eleven high-impact journals were searched from 2009 to 2019. The included studies were assessed on the basis of 16 domains. Seven domains were deemed critical for high-quality studies. On the basis of the performance in these 16 domains with different weights, overall ratings were generated, and the quality was determined to be "high," "moderate," "low," or "critically low."

RESULTS

Eighty-one heart failure-related SRs with MAs were included. Overall, 79 studies were of "critically low quality" and two were of "low quality." These findings were attributed to insufficiency in the following critical domains: a priori protocols (compliance rate, 5%), complete list of exclusions with justification (5%), risk of bias assessment (69%), meta-analysis methodology (78%), and investigation of publication bias (60%).

CONCLUSIONS

The low ratings for these potential high-quality heart failure-related SRs and MAs challenge the discrimination capacity of AMSTAR 2. In addition to identifying certain areas of insufficiency, these findings indicate the need to justify or modify AMSTAR 2's rating rules.

Digital Technologies for Health Promotion and Disease Prevention in Older People: Protocol for a Scoping Review

BACKGROUND

Digital technologies could contribute to health promotion and disease prevention. It is unclear if and how such digital technologies address the health needs of older people in nonclinical settings (ie, daily life).

OBJECTIVE

This study aims to identify digital technologies for health promotion and disease prevention that target the needs of older people in nonclinical settings by performing a scoping review of the published literature. The scoping review is guided by the framework of Arksey and O'Malley.

METHODS

Our scoping review follows the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. The information sources are bibliographic databases (MEDLINE, PsycINFO, CINAHL, and SCOPUS) and bibliographies of any included systematic reviews. Manual searches for additional studies will be performed in Google Scholar and most relevant journals. The electronic search strategy was developed in collaboration with a librarian who performed the search for studies on digital technologies for health promotion and disease prevention targeting the needs of older people. Study selection and data coding will be performed independently by 2 authors. Consensus will be reached by discussion. Eligibility is based on the PCC (Population, Concept, and Context) criteria as follows: (1) older people (population); (2) any digital (health) technology, such as websites, smartphone apps, or wearables (concept); and (3) health promotion and disease prevention in nonclinical (daily life, home, or community) settings (context). Primary studies with any design or reviews with a systematic methodology published in peer-reviewed academic journals will be included. Data items will address study designs, PCC criteria, benefits or barriers related to digital technology use by older people, and evidence gaps. Data will be synthesized using descriptive statistics or narratively described by identifying common themes. Quality appraisal will be performed for any included systematic reviews, using a validated instrument for this study type (A Measurement Tool to Assess Systematic Reviews, version 2 [AMSTAR2]).

RESULTS

Following preliminary literature searches to test and calibrate the search syntax, the electronic literature search was performed in March 2022 and manual searches were completed in June 2022. Study selection based on titles and abstracts was completed in July 2022, and the full-text screen was initiated in July 2022.

CONCLUSIONS

Our scoping review will identify the types of digital technologies, health targets in the context of health promotion and disease prevention, and health benefits or barriers associated with the use of such technologies for older people in nonclinical settings. This knowledge could guide further research on how digital technologies can support healthy aging.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/37729.

Evaluation of Digital Interventions for Physical Activity Promotion: Scoping Review

Background

Digital interventions are interventions supported by digital tools or technologies, such as mobile apps, wearables, or web-based software. Digital interventions in the context of public health are specifically designed to promote and improve health. Recent reviews have shown that many digital interventions target physical activity promotion; however, it is unclear how such digital interventions are evaluated.

Objective

We aimed to investigate evaluation strategies in the context of digital interventions for physical activity promotion using a scoping review of published reviews. We focused on the target (ie, user outcomes or tool performance), methods (ie, tool data or self-reported data), and theoretical frameworks of the evaluation strategies.

Methods

A protocol for this study was preregistered and published. From among 300 reviews published up to March 19, 2021 in Medline, PsycINFO, and CINAHL databases, 40 reviews (1 rapid, 9 scoping, and 30 systematic) were included in this scoping review. Two authors independently performed study selection and data coding. Consensus was reached by discussion. If applicable, data were coded quantitatively into predefined categories or qualitatively using definitions or author statements from the included reviews. Data were analyzed using either descriptive statistics, for quantitative data (relative frequencies out of all studies), or narrative synthesis focusing on common themes, for qualitative data.

Results

Most reviews that were included in our scoping review were published in the period from 2019 to 2021 and originated from Europe or Australia. Most primary studies cited in the reviews included adult populations in clinical or nonclinical settings, and focused on mobile apps or wearables for physical activity promotion. The evaluation target was a user outcome (efficacy, acceptability, usability, feasibility, or engagement) in 38 of the 40 reviews or tool performance in 24 of the 40 reviews. Evaluation methods relied upon objective tool data (in 35/40 reviews) or other data from self-reports or assessments (in 28/40 reviews). Evaluation frameworks based on behavior change theory, including goal setting, self-monitoring, feedback on behavior, and educational or motivational content, were mentioned in 22 out of 40 reviews. Behavior change theory was included in the development phases of digital interventions according to the findings of 20 out of 22 reviews.

Conclusions

The evaluation of digital interventions is a high priority according to the reviews included in this scoping review. Evaluations of digital interventions, including mobile apps or wearables for physical activity promotion, typically target user outcomes and rely upon objective tool data. Behavior change theory may provide useful guidance not only for development of digital interventions but also for the evaluation of user outcomes in the context of physical activity promotion. Future research should investigate factors that could improve the efficacy of digital interventions and the standardization of terminology and reporting in this field.

International Registered Report Identifier (IRRID)

RR2-10.2196/35332

Development of a Framework for the Implementation of Synchronous Digital Mental Health: Realist Synthesis of Systematic Reviews

BACKGROUND

The use of technologies has served to reduce gaps in access to treatment, and digital health interventions show promise in the care of mental health problems. However, to understand what and how these interventions work, it is imperative to document the aspects related to their challenging implementation.

OBJECTIVE

The aim of this study was to determine what evidence is available for synchronous digital mental health implementation and to develop a framework, informed by a realist review, to explain what makes digital mental health interventions work for people with mental health problems.

METHODS

The SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, and Research type) framework was used to develop the following review question: What makes digital mental health interventions with a synchronous component work on people with mental health problems, including depression, anxiety, or stress, based on implementation, economic, quantitative, qualitative, and mixed methods studies? The MEDLINE, EBM Reviews, PsycINFO, EMBASE, SCOPUS, CINAHL Complete, and Web of Science databases were searched from January 1, 2015, to September 2020 with no language restriction. A Measurement Tool to Assess Systematic Reviews-2 (AMSTAR-2) was used to assess the risk of bias and Confidence in Evidence from Reviews of Qualitative Research (CERQual) was used to assess the confidence in cumulative evidence. Realist synthesis analysis allowed for developing a framework on the implementation of synchronous digital mental health using a grounded-theory approach with an emergent approach.

RESULTS

A total of 21 systematic reviews were included in the study. Among these, 90% (n=19) presented a critically low confidence level as assessed with AMSTAR-2. The realist synthesis allowed for the development of three hypotheses to identify the context and mechanisms in which these interventions achieve these outcomes: (1) these interventions reach populations otherwise unable to have access because they do not require the physical presence of the therapist nor the patient, thereby tackling geographic barriers posed by in-person therapy; (2) these interventions reach populations otherwise unable to have access because they can be successfully delivered by nonspecialists, which makes them more cost-effective to implement in health services; and (3) these interventions are acceptable and show good results in satisfaction because they require less need of disclosure and provide more privacy, comfortability, and participation, enabling the establishment of rapport with the therapist.

CONCLUSIONS

We developed a framework with three hypotheses that explain what makes digital mental health interventions with a synchronous component work on people with mental health problems. Each hypothesis represents essential outcomes in the implementation process.

TRIAL REGISTRATION

PROSPERO International Prospective Register of Systematic Reviews CRD42020203811; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020203811.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.12688/f1000research.27150.2.

Evaluation of Digital Interventions for Physical Activity Promotion: Protocol for a Scoping Review

BACKGROUND

Digital interventions (DIs) could support physical activity (PA) promotion, according to recent reviews. However, it remains unclear if and how DIs for PA promotion are evaluated; thus, it is unclear if they support behavior change in real-world settings. A mapping of evidence from published reviews is required to focus on the evaluation of DIs for PA promotion.

OBJECTIVE

The aim of our study is to investigate evaluation strategies for any outcome in the context of DIs for PA promotion by conducting a scoping review of published reviews.

METHODS

Our scoping review adheres to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. The information sources include bibliographic databases (MEDLINE, PsycINFO, and CINAHL) and the bibliographies of the selected studies. The electronic search strategy was developed and conducted in collaboration with an experienced database specialist. The electronic search was conducted in English with no limits up to March 19, 2021, for sources with the terms digital intervention AND evaluation AND physical activity in titles or abstracts. After deduplication, 300 reviews selected from 4912 search results were assessed for eligibility by 2 authors working independently. The inclusion criteria were (1) healthy or clinical samples (population), (2) DIs for PA promotion (intervention), (3) comparisons to any other intervention or no intervention (comparison), (4) evaluation strategies (methods, results, or frameworks) for any outcome in the context of DIs for PA promotion (outcome), and (5) any published review (study type). According to the consensus reached during a discussion, 40 reviews met the inclusion criteria-36 from the electronic search and 4 from the manual search of the bibliographies of the 36 reviews. All reviews reported the evaluation strategies for any outcomes in the context of DIs for PA promotion in healthy or clinical samples. Data coding and the quality appraisal of systematic reviews are currently being performed independently by 2 authors.

RESULTS

Our scoping review includes data from 40 published reviews (1 rapid review, 9 scoping reviews, and 30 systematic reviews). The focus of data coding is on evaluation strategies in the context of DIs for PA promotion and on the critical appraisal of the included systematic reviews. The final consensus regarding all data is expected in early 2022.

CONCLUSIONS

Interventions for PA promotion that are supported by digital technologies require evaluation to ensure their efficacy in real-world settings. Our scoping review is needed because it addresses novel objectives that focus on such evaluations and are not answered in the published reviews identified in our search. The evaluation strategies addressing DIs for PA promotion will be mapped to synthesize the results that have been reported in published reviews so far.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/35332.

An Evaluation of Evidence Underpinning Management Recommendations in Tobacco Use Disorder Clinical Practice Guidelines

INTRODUCTION

Clinical practice guidelines(CPGs) are important tools for medical decision-making. Given the high prevalence and financial burden associated with tobacco use disorder(TUD), it is critical that recommendations within CPGs are based on robust evidence. Systematic reviews(SRs) are considered the highest level of evidence, thus, we evaluated the quality of SRs underpinning CPG recommendations for TUD.

METHODS

We used PubMed to search for CPGs relating to TUD published between January 1, 2010 and May 21, 2021. SRs were extracted from CPG references and evaluated using Preferred Reporting Items for Systematic Reviews and Meta-Analyses(PRISMA) and A MeaSurement Tool to Assess Systematic Reviews(AMSTAR-2) tools. We then compared SRs conducted by the Cochrane Collaboration with non-Cochrane SRs using a Mann-Whitney U test and determined associations between PRISMA and AMSTAR-2 extracted characteristics using multiple regression.

RESULTS

Our search generated 10 CPGs with 98 SRs extracted. Mean PRISMA completion was 74.7%(SD = 16.7) and mean AMSTAR-2 completion was 53.8%(SD = 22.0) across all guidelines. Cochrane SRs were more complete than non-Cochrane studies in the PRISMA and AMSTAR-2 assessments. The regression model showed a statistically significant association between PRISMA completion and AMSTAR-2 rating, with those classified as "low" or "moderate" quality having higher PRISMA completion than those with "critically low" ratings.

CONCLUSION

We found substandard adherence to PRISMA and AMSTAR-2 checklists across SRs cited in TUD CPGs. A lack of recent SRs in CPGs could lead to outdated recommendations. Therefore, frequent guideline updates with recently published evidence may ensure more accurate clinical recommendations and improve patient care.

IMPLICATIONS

Systematic reviews used to underpin clinical practice guideline recommendations influence treatment decisions and, ultimately, patient outcomes. We found that many systematic reviews underpinning tobacco use disorder guideline recommendations were out of date and unsatisfactory in reporting and quality. Thus, including newer systematic reviews containing more recently conducted trials and better reporting could alter recommendations and improve the rate of successful tobacco cessation attempts.

Predictors of Higher Quality of Systematic Reviews Addressing Nutrition and Cancer Prevention

Systematic reviews/meta-analyses (SR/MAs) are considered a reliable source of information in healthcare. We aimed to explore the association of several characteristics of SR/MAs addressing nutrition in cancer prevention and their quality/risk of bias (using assessments from AMSTAR-2 and ROBIS tools). The analysis included 101 SR/MAs identified in a systematic survey. Associations of each specified characteristic (e.g., information about the protocol, publication year, reported use of GRADE, or other methods for assessing overall certainty of evidence) with the number of AMSTAR-2 not met (‘No’ responses) and the number of ROBIS items met (‘Probably Yes’ or “Yes’ responses) were examined. Poisson regression was used to identify predictors of the number of ‘No’ answers (indicating lower quality) for all AMSTAR-2 items and the number of ‘Yes’ or ‘Probably Yes’ answers (indicating higher quality/lower concern for bias) for all ROBIS items. Logistic regression was used to identify variables associated with at least one domain assessed as ‘low concern for bias’ in the ROBIS tool. In multivariable analysis, SR/MAs not reporting use of any quality/risk of bias assessment instrument for primary studies were associated with a higher number of ‘No’ answers for all AMSTAR-2 items (incidence rate ratio (IRR) 1.26, 95% confidence interval (CI) 1.09–1.45), and a lower number of ‘Yes’ or ‘Probably Yes’ answers for all ROBIS items (IRR 0.76, 95% CI 0.66–0.87). Providing information about the protocol and search for unpublished studies was associated with a lower number of ‘No’ answers (IRR 0.73, 95% CI 0.56–0.97 and IRR 0.75, 95% CI 0.59–0.95, respectively) and a higher number of ‘Yes’ or ‘Probably Yes’ answers (IRR 1.43, 95% CI 1.17–1.74 and IRR 1.28, 95% CI 1.07–1.52, respectively). Not using at least one quality/risk of bias assessment tool for primary studies within an SR/MA was associated with lower odds that a study would be assessed as ‘low concern for bias’ in at least one ROBIS domain (odds ratio 0.061, 95% CI 0.007–0.527). Adherence to methodological standards in the development of SR/MAs was associated with a higher overall quality of SR/MAs addressing nutrition for cancer prevention.

Cochrane's risk of bias tool for non-randomized studies (ROBINS-I) is frequently misapplied: A methodological systematic review

OBJECTIVES

We aimed to review how 'Risk of Bias In Non-randomized Studies-of Interventions' (ROBINS-I), a Cochrane risk of bias assessment tool, has been used in recent systematic reviews.

STUDY DESIGN AND SETTING

Database and citation searches were conducted in March 2020 to identify recently published reviews using ROBINS-I. Reported ROBINS-I assessments and data on how ROBINS-I was used were extracted from each review. Methodological quality of reviews was assessed using AMSTAR 2 ('A MeaSurement Tool to Assess systematic Reviews').

RESULTS

Of 181 hits, 124 reviews were included. Risk of bias was serious/critical in 54% of assessments on average, most commonly due to confounding. Quality of reviews was mostly low, and modifications and incorrect use of ROBINS-I were common, with 20% reviews modifying the rating scale, 20% understating overall risk of bias, and 19% including critical-risk of bias studies in evidence synthesis. Poorly conducted reviews were more likely to report low/moderate risk of bias (predicted probability 57% [95% CI: 47-67] in critically low-quality reviews, 31% [19-46] in high/moderate-quality reviews).

CONCLUSION

Low-quality reviews frequently apply ROBINS-I incorrectly, and may thus inappropriately include or give too much weight to uncertain evidence. Readers should be aware that such problems can lead to incorrect conclusions in reviews.

Use of ketamine and esketamine for depression: an overview of systematic reviews with meta-analyses

PURPOSE

To summarize the evidence of efficacy and safety of the use of ketamine and esketamine for depression.

METHODS

A literature search was performed in Medline, the Cochrane Library, LILACS, and CRD until November 2020. We included systematic reviews with meta-analyses of randomized controlled trials on the use of ketamine and esketamine in adult patients with depression. Two authors independently performed the study selection and data extraction. The AMSTAR-2 tool was used to appraise the quality of included reviews.

RESULTS

A total of 118 records were identified, and 11 studies fully met the eligibility criteria. Compared to control, ketamine improved the clinical response at 40 min to 1 week and clinical remission at 80 min to 72 h, and esketamine improved both outcomes at 2 h to 4 weeks. Ketamine and esketamine also had a beneficial effect on the depression scales score and suicidality. For adverse events, oral ketamine did not show significant change compared to control, while intranasal esketamine showed difference for any events, such as dissociation, dizziness, hypoesthesia, and vertigo. Most reviews were classified as "critically low quality," and none of them declared the source of funding of the primary studies and assessed the potential impact of risk of bias in primary studies.

CONCLUSION

Ketamine and esketamine showed a significant antidepressant action within a few hours or days after administration; however, the long-term efficacy and safety are lacking. In addition, the methodological quality of the reviews was usually critically low, which may indicate the need for higher quality evidence in relation to the theme.

The application of AMSTAR2 in 32 overviews of systematic reviews of interventions for mental and behavioural disorders: A cross-sectional study

'A measurement tool to assess systematic reviews, version 2' (AMSTAR2) is a 16-item tool to critically appraise systematic reviews (SRs) of healthcare interventions. This study aimed to assess the methods and outcomes of AMSTAR2 appraisals in overviews of SRs of interventions for mental and behavioural disorders. The cross-sectional study was conducted using 32 overviews of SRs selected from three electronic databases in January 2021. Data items included overview and SR characteristics and AMSTAR2 appraisal methods and outcomes. Data were extracted by two authors independently and narratively synthesised using descriptive statistics (means ± SD and relative frequencies). SR characteristics were compared based on AMSTAR2 appraisal outcomes using chi-square tests. The 32 overviews appraised SRs of predominantly non-pharmacological interventions for mental disorders. AMSTAR2 appraisals were reported as confidence ratings in 25/32 overviews or individual item scores in 24/32 overviews. Most SRs/overview were non-Cochrane (mean = 94%), included RCTs only (mean = 77%) and were published before AMSTAR2 release (mean = 79%). The confidence ratings derived in 25 overviews for 349 SRs were predominantly critically low (68%). Confidence ratings were similar for SRs with RCTs only versus RCTs+non-RCTs or SRs published before versus after AMSTAR2 release, while Cochrane SRs received more high+moderate than low+critically low confidence ratings (p < 0.01). Confidence ratings derived based on AMSTAR2 do not differentiate among SRs of healthcare interventions except for Cochrane SRs that fulfil the criteria for high confidence ratings. AMSTAR2 items should be consulted to avoid common weaknesses in future SRs.

The methodological quality and clinical applicability of meta-analyses on probiotics in 2020: A cross-sectional study

Systematic reviews with meta-analyses (SR/MA) are frequently conducted to investigate clinical efficacy of probiotics. However, only rigorously prepared analyses can serve as the highest level of evidence for a specified research question. We have aimed to determine (1) what is the methodological quality of recent SR/MA conducted to assess the efficacy of probiotics; (2) whether the results of SR/MA have a clinical application; and (3) what are factors associated with better quality and applicability of the SR/MA. We systematically searched 4 databases for SR/MA on the probiotics efficacy published in 2020 (PROSPERO CRD42020222716). The AMSTAR 2 tool and pre-defined authors' criteria were used to evaluate methodological quality and clinical applicability, respectively. A total of 114 SR/MA were appraised. In the case of 88 papers (77%), the overall confidence in the results was rated as "critically low". The most prevalent flaws were lack of list of excluded studies with justification (79.8%), lack of study protocol (60.5%), and problems with appropriate results combination(54.4%). A declaration of conduction a probiotic efficacy SR/MA could have been misleading in case of 18 studies that included also synbiotics, paraprobiotics, and prebiotics trials in analyses. Only 14 SR/MA provided results that can be apply in clinical practice. Higher journal impact factor and European affiliation of the 1st and corresponding authors were most consistently associated with higher odds of AMSTAR 2 items fulfillments. Based on our findings, SR/MA of probiotics trials cannot be treated as the highest level of evidence without a careful evaluation of their methodological validity.