Development of the Documentation and Appraisal Review Tool for systematic reviews

Critical appraisal of methodological quality and completeness of reporting in Chinese social science systematic reviews with meta-analysis: A systematic review

Background

A systematic review is a type of literature review that uses rigorous methods to synthesize evidence from multiple studies on a specific topic. It is widely used in academia, including medical and social science research. Social science is an academic discipline that focuses on human behaviour and society. However, consensus regarding the standards and criteria for conducting and reporting systematic reviews in social science is lacking. Previous studies have found that the quality of systematic reviews in social science varies depending on the topic, database, and country.

Objectives

This study evaluates the completeness of reporting and methodological quality of intervention and non-intervention systematic reviews in social science in China. Additionally, we explore factors that may influence quality.

Search Methods

We searched three major Chinese electronic databases-CNKI, VIP, and Wangfang-for intervention and non-intervention reviews in social science published in Chinese journals from 1 January 2009 to 2 December 2022.

Selection Criteria

We included intervention and non-intervention reviews; however, we excluded overviews, qualitative syntheses, integrative reviews, rapid reviews, and evidence syntheses/summaries. We also excluded meta-analyses that used advanced methods (e.g., cross-sectional, cumulative, Bayesian, structural equation, or network meta-analyses) or that focused on instrument validation.

Data Collection and Analysis

We extracted data using a coding form with publication information and study content characteristics. This study conducted pilot extraction and quality assessment with four authors and formal extraction and assessment with two groups of four authors each. PRISMA2020 and MOOSE were used to evaluate the reporting completeness of intervention and non-intervention reviews. AMSTAR-2 and DART tools were adopted to assess their methodological quality. We described the characteristics of the included reviews with frequencies and percentages. We used SPSS (version 26.0) to conduct a linear regression analysis and ANOVA to explore the factors that may influence both completeness of reporting and methodological quality.

Main Results

We included 1176 systematic reviews with meta-analyses published in Chinese journals between 2009 and 2022. The top three fields of publication were psychology (417, 35.5%), education (388, 33.0%), and management science (264, 22.4%). Four hundred and thirty-two intervention reviews were included. The overall completeness of reporting in PRISMA and compliance rate of the methodological process in AMSTAT-2 were 49.9% and 45.5%, respectively. Intervention reviews published in Chinese Social Science Citation Index (CSSCI) journals had lower reporting completeness than those published in non-CSSCI journals (46.7% vs. 51.1%), similar to methodological quality (39.6% vs. 47.9%). A few reviews reported the details on registration (0.2%), rationality of study selection criteria (1.6%), sources of funding for primary studies (0.2%), reporting bias assessment (2.8%), certainty of evidence assessment (1.2%), and sensitivity analysis (107, 24.8%). Seven hundred and forty-four non-intervention reviews were included. The overall completeness of reporting in MOOSE and compliance rate of the methodological process in DART were 51.8% and 50.5%, respectively. Non-intervention reviews published in CSSCI journals had higher reporting completeness than those published in non-CSSCI journals (53.3% vs. 50.3%); however, there was no difference in methodological quality (51.0% vs. 50.0%). Most reviews did not report the process and results of selection (80.8%), and 58.9% of reviews did not describe the process of data extraction; only 9.5% assessed the quality of included studies; while none of the reviews examined bias by confounding, outcome reporting bias, and loss to follow-up. An improving trend over time was observed for both intervention and non-intervention reviews in completeness of reporting and methodological quality (PRISMA: β = 0.24, p < 0.01; AMSTAR-2: β = 0.17, p < 0.01; MOOSE: β = 0.34, p < 0.01; DART: β = 0.30, p < 0.01). The number of authors and financial support also have a positive effect on quality.

Authors' Conclusions

Completeness of reporting and methodological quality were low in both intervention and non-intervention reviews in Chinese social sciences, especially regarding registration, protocol, risk of bias assessment, and data and code sharing. The sources of literature, number of authors, publication year, and funding source declarations were identified as factors that may influence the quality of reviews. More rigorous standards and guidelines for conducting and reporting reviews are required in social science research as well as more support and incentives for reviewers to adhere to them.

Respiratory Management of Patients With Neuromuscular Weakness: An American College of Chest Physicians Clinical Practice Guideline and Expert Panel Report

BACKGROUND

Respiratory failure is a significant concern in neuromuscular diseases (NMDs). This CHEST guideline examines the literature on the respiratory management of patients with NMD to provide evidence-based recommendations.

STUDY DESIGN AND METHODS

An expert panel conducted a systematic review addressing the respiratory management of NMD and applied the Grading of Recommendations, Assessment, Development, and Evaluations approach for assessing the certainty of the evidence and formulating and grading recommendations. A modified Delphi technique was used to reach a consensus on the recommendations.

RESULTS

Based on 128 studies, the panel generated 15 graded recommendations, one good practice statement, and one consensus-based statement.

INTERPRETATION

Evidence of best practices for respiratory management in NMD is limited and is based primarily on observational data in amyotrophic lateral sclerosis. The panel found that pulmonary function testing every 6 months may be beneficial and may be used to initiate noninvasive ventilation (NIV) when clinically indicated. An individualized approach to NIV settings may benefit patients with chronic respiratory failure and sleep-disordered breathing related to NMD. When resources allow, polysomnography or overnight oximetry can help to guide the initiation of NIV. The panel provided guidelines for mouthpiece ventilation, transition to home mechanical ventilation, salivary secretion management, and airway clearance therapies. The guideline panel emphasizes that NMD pathologic characteristics represent a diverse group of disorders with differing rates of decline in lung function. The clinician's role is to add evaluation at the bedside to shared decision-making with patients and families, including respect for patient preferences and treatment goals, considerations of quality of life, and appropriate use of available resources in decision-making.

NICU sensory experiences associated with positive outcomes: an integrative review of evidence from 2015-2020

To inform changes to the Supporting and Enhancing NICU Sensory Experiences (SENSE) program, studies investigating sensory-based interventions in the NICU with preterm infants born ≤32 weeks were identified. Studies published between October 2015 to December 2020, and with outcomes related to infant development or parent well-being, were included in this integrative review. The systematic search used databases including MEDLINE, Cumulative Index to Nursing and Allied Health Literature, the Cochrane Library, and Google Scholar. Fifty-seven articles (15 tactile, 9 auditory, 5 visual, 1 gustatory/olfactory, 5 kinesthetic, and 22 multimodal) were identified. The majority of the sensory interventions that were identified within the articles were reported in a previous integrative review (1995-2015) and already included in the SENSE program. New evidence has led to refinements of the SENSE program, notably the addition of position changes across postmenstrual age (PMA) and visual tracking starting at 34 weeks PMA.

CAT HPPR: a critical appraisal tool to assess the quality of systematic, rapid, and scoping reviews investigating interventions in health promotion and prevention

BACKGROUND

For over three decades researchers have developed critical appraisal tools (CATs) for assessing the scientific quality of research overviews. Most established CATs for reviews in evidence-based medicine and evidence-based public health (EBPH) focus on systematic reviews (SRs) with studies on experimental interventions or exposure included. EBPH- and implementation-oriented organisations and decision-makers, however, often seek access to rapid reviews (RRs) or scoping reviews (ScRs) for rapid evidence synthesis and research field exploration. Until now, no CAT is available to assess the quality of SRs, RRs, and ScRs following a unified approach. We set out to develop such a CAT.

METHODS

The development process of the Critical Appraisal Tool for Health Promotion and Prevention Reviews (CAT HPPR) included six phases: (i) the definition of important review formats and complementary approaches, (ii) the identification of relevant CATs, (iii) prioritisation, selection and adaptation of quality criteria using a consensus approach, (iv) development of the rating system and bilingual guidance documents, (v) engaging with experts in the field for piloting/optimising the CAT, and (vi) approval of the final CAT. We used a pragmatic search approach to identify reporting guidelines/standards (n = 3; e.g. PRISMA, MECIR) as well as guidance documents (n = 17; e.g. for reviews with mixed-methods approach) to develop working definitions for SRs, RRs, ScRs, and other review types (esp. those defined by statistical methods or included data sources).

RESULTS

We successfully identified 14 relevant CATs, predominantly for SRs (e.g. AMSTAR 2), and extracted 46 items. Following consensual discussions 15 individual criteria were included in our CAT and tailored to the review types of interest. The CAT was piloted with 14 different reviews which were eligible to be included in a new German database looking at interventions in health promotion and prevention in different implementation settings.

CONCLUSIONS

The newly developed CAT HPPR follows a unique uniformed approach to assess a set of heterogeneous reviews (e.g. reviews from problem identification to policy evaluations) to assist end-users needs. Feedback of external experts showed general feasibility and satisfaction with the tool. Future studies should further formally test the validity of CAT HPPR using larger sets of reviews.

PROTOCOL: Critical appraisal of methodological quality and reporting items of systematic reviews with meta-analysis in evidence-based social science in China: A systematic review

This is the protocol for a Campbell systematic review. The objectives are as follows: (1) To evaluate the reporting quality of systematic reviews published in Chinese social science journals against the PRISMA and MOOSE standards; (2) To evaluate the methodology quality of systematic reviews published in Chinese social science journals against the AMSTAR-2 and DART standards; and (3) To analyze other characteristics of systematic reviews published in Chinese social science journals using content analysis.

Diagnosis and Evaluation of Hypersensitivity Pneumonitis: CHEST Guideline and Expert Panel Report

BACKGROUND

The purpose of this analysis is to provide evidence-based and consensus-derived guidance for clinicians to improve individual diagnostic decision-making for hypersensitivity pneumonitis (HP) and decrease diagnostic practice variability.

STUDY DESIGN AND METHODS

Approved panelists developed key questions regarding the diagnosis of HP using the PICO (Population, Intervention, Comparator, Outcome) format. MEDLINE (via PubMed) and the Cochrane Library were systematically searched for relevant literature, which was supplemented by manual searches. References were screened for inclusion, and vetted evaluation tools were used to assess the quality of included studies, to extract data, and to grade the level of evidence supporting each recommendation or statement. The quality of the evidence was assessed using the GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) approach. Graded recommendations and ungraded consensus-based statements were drafted and voted on using a modified Delphi technique to achieve consensus. A diagnostic algorithm is provided, using supporting data from the recommendations where possible, along with expert consensus to help physicians gauge the probability of HP.

RESULTS

The systematic review of the literature based on 14 PICO questions resulted in 14 key action statements: 12 evidence-based, graded recommendations and 2 ungraded consensus-based statements. All evidence was of very low quality.

INTERPRETATION

Diagnosis of HP should employ a patient-centered approach and include a multidisciplinary assessment that incorporates the environmental and occupational exposure history and CT pattern to establish diagnostic confidence prior to considering BAL and/or lung biopsy. Criteria are presented to facilitate diagnosis of HP. Additional research is needed on the performance characteristics and generalizability of exposure assessment tools and traditional and new diagnostic tests in modifying clinical decision-making for HP, particularly among those with a provisional diagnosis.

Chronic Cough Due to Stable Chronic Bronchitis: CHEST Expert Panel Report

BACKGROUND

Chronic cough due to chronic bronchitis (CB) causes significant impairment in quality of life, and effective treatment strategies are needed. We conducted a systematic review on the management of chronic cough due to CB to update the recommendations and suggestions of the American College of Chest Physicians (CHEST) 2006 guideline on this topic.

METHODS

This systematic review asked three questions: (1) What are the clinical features of the history that suggest a patient's cough-phlegm syndrome is due to CB? (2) Can treatment of stable CB improve or eliminate chronic cough? (3) Can therapy that targets chronic cough due to CB prevent or reduce the occurrence of acute CB exacerbations? Studies of adult patients with CB were included and assessed for relevance and quality. Based on the systematic review, guideline suggestions were developed and voted on by using the CHEST organization methodology.

RESULTS

The search strategy used an assortment of descriptors and assessments to identify studies of chronic cough due to CB.

CONCLUSIONS

The evidence supporting the management of chronic cough due to CB is limited overall and of low quality. This article provides guidance on treatment by presenting suggestions based on the best currently available evidence and identifies gaps in our knowledge and areas for future research.

Managing Chronic Cough Due to Asthma and NAEB in Adults and Adolescents: CHEST Guideline and Expert Panel Report

BACKGROUND

Asthma and non-asthmatic eosinophilic bronchitis (NAEB) are among the commonest causes of chronic cough in adults. We sought to determine the role of non-invasive measurements of airway inflammation, including induced sputum and fractional exhaled nitric oxide, in the evaluation of cough associated with asthma, and what the best treatment is for cough due to asthma or NAEB.

METHODS

We undertook three systematic reviews of randomized controlled trials and observational trials of adults and adolescents > 12 years of age with a chronic cough due to asthma or NAEB. Eligible studies were identified in MEDLINE, CENTRAL, and SCOPUS and assessed for relevance and quality. Guidelines were developed and voted upon using CHEST guideline methodology.

RESULTS

Of the citations reviewed, 3/1,175, 53/656, and 6/134 were identified as being eligible for inclusion in the three systematic reviews, respectively. In contrast to established guidelines for asthma therapies in general and the inclusion in some guidelines for a role of biomarkers of airway inflammation to guide treatment in severe disease, the evidence of specific benefit related to the use of non-invasive biomarkers in patients with chronic cough due to asthma was weak. The best therapeutic option for cough in asthma or NAEB is inhaled corticosteroids followed by leukotriene receptor antagonism.

CONCLUSIONS

This guideline offers recommendations on the role of non-invasive measurements of airway inflammation and treatment for cough due to asthma or NAEB based on the available literature, and identifies gaps in knowledge and areas for future research.

Clinically Diagnosing Pertussis-associated Cough in Adults and Children: CHEST Guideline and Expert Panel Report

BACKGROUND

The decision to treat a suspected case of pertussis with antibiotics is usually based on a clinical diagnosis rather than waiting for laboratory confirmation. The current guideline focuses on making the clinical diagnosis of pertussis-associated cough in adults and children.

METHODS

The American College of Chest Physicians (CHEST) methodologic guidelines and the Grading of Recommendations, Assessment, Development, and Evaluation framework were used. The Expert Cough Panel based their recommendations on findings from a systematic review that was recently published on the topic; final grading was reached by consensus according to Delphi methodology. The systematic review was carried out to answer the Key Clinical Question: In patients presenting with cough, how can we most accurately diagnose from clinical features alone those who have pertussis-associated cough as opposed to other causes of cough?

RESULTS

In adults, after pre-specified meta-analysis exclusions, pooled estimates of sensitivity and specificity were generated for only 4 clinical features: paroxysmal cough, post-tussive vomiting, inspiratory whooping, and absence of fever. Both paroxysmal cough and absence of fever had high sensitivity (93.2% [95% CI, 83.2-97.4] and 81.8% [95% CI, 72.2-88.7], respectively) and low specificity (20.6% [95% CI, 14.7-28.1] and 18.8% [95% CI, 8.1-37.9]). Inspiratory whooping and posttussive vomiting had a low sensitivity (32.5% [95% CI, 24.5-41.6] and 29.8% [95% CI, 18.0-45.2]) but high specificity (77.7% [95% CI, 73.1-81.7] and 79.5% [95% CI, 69.4-86.9]). In children, after pre-specified meta-analysis exclusions, pooled estimates of sensitivity and specificity were generated for only 1 clinical feature in children (0-18 years): posttussive vomiting. Posttussive vomiting in children was only moderately sensitive (60.0% [95% CI, 40.3-77.0]) and specific (66.0% [95% CI, 52.5-77.3]).

CONCLUSIONS

In adults with acute (< 3 weeks) or subacute (3-8 weeks) cough, the presence of whooping or posttussive vomiting should rule in a possible diagnosis of pertussis, whereas the lack of a paroxysmal cough or the presence of fever should rule it out. In children with acute (< 4 weeks) cough, posttussive vomiting is suggestive of pertussis but is much less helpful as a clinical diagnostic test. Guideline suggestions are made based upon these findings and conclusions.

Treating Cough Due to Non-CF and CF Bronchiectasis With Nonpharmacological Airway Clearance: CHEST Expert Panel Report

BACKGROUND

In bronchiectasis due to cystic fibrosis (CF) and other causes, airway clearance is one of the mainstays of management. We conducted a systematic review on airway clearance by using non-pharmacological methods as recommended by international guidelines to develop recommendations or suggestions to update the 2006 CHEST guideline on cough.

METHODS

The systematic search for evidence examined the question, "Is there evidence of clinically important treatment effects for non-pharmacological therapies in cough treatment for patients with bronchiectasis?" Populations selected were all patients with bronchiectasis due to CF or non-CF bronchiectasis. The interventions explored were the non-pharmacological airway clearance therapies. The comparison populations included those receiving standard therapy and/or placebo. Clinically important outcomes that were explored were exacerbation rates, quality of life, hospitalizations, and mortality.

RESULTS

In both CF and non-CF bronchiectasis, there were systematic reviews and overviews of systematic reviews identified. Despite these findings, there were no large randomized controlled trials that explored the impact of airway clearance on exacerbation rates, quality of life, hospitalizations, or mortality.

CONCLUSIONS

Although the cough panel was not able to make recommendations, they have made consensus-based suggestions and provided direction for future studies to fill the gaps in knowledge.

Screening for Lung Cancer: CHEST Guideline and Expert Panel Report

BACKGROUND

Low-dose chest CT screening for lung cancer has become a standard of care in the United States in the past few years, in large part due to the results of the National Lung Screening Trial. The benefit and harms of low-dose chest CT screening differ in both frequency and magnitude. The translation of a favorable balance of benefit and harms into practice can be difficult. Here, we update the evidence base for the benefit, harms, and implementation of low radiation dose chest CT screening. We use the updated evidence base to provide recommendations where the evidence allows, and statements based on experience and expert consensus where it does not.

METHODS

Approved panelists developed key questions using the PICO (population, intervention, comparator, and outcome) format to address the benefit and harms of low-dose CT screening, as well as key areas of program implementation. A systematic literature review was conducted by using MEDLINE via PubMed, Embase, and the Cochrane Library. Reference lists from relevant retrievals were searched, and additional papers were added. The quality of the evidence was assessed for each critical or important outcome of interest using the GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) approach. Important clinical questions were addressed based on the evidence developed from the systematic literature review. Graded recommendations and ungraded statements were drafted, voted on, and revised until consensus was reached.

RESULTS

The systematic literature review identified 59 studies that informed the response to the 12 PICO questions that were developed. Key clinical questions were addressed resulting in six graded recommendations and nine ungraded consensus based statements.

CONCLUSIONS

Evidence suggests that low-dose CT screening for lung cancer results in a favorable but tenuous balance of benefit and harms. The selection of screen-eligible patients, the quality of imaging and image interpretation, the management of screen-detected findings, and the effectiveness of smoking cessation interventions can affect this balance. Additional research is needed to optimize the approach to low-dose CT screening.

Classification of Cough as a Symptom in Adults and Management Algorithms: CHEST Guideline and Expert Panel Report

BACKGROUND

We performed systematic reviews using the population, intervention, comparison, outcome (PICO) format to answer the following key clinical question: Are the CHEST 2006 classifications of acute, subacute and chronic cough and associated management algorithms in adults that were based on durations of cough useful?

METHODS

We used the CHEST Expert Cough Panel's protocol for the systematic reviews and the American College of Chest Physicians (CHEST) methodological guidelines and Grading of Recommendations Assessment, Development, and Evaluation framework. Data from the systematic reviews in conjunction with patient values and preferences and the clinical context were used to form recommendations or suggestions. Delphi methodology was used to obtain the final grading.

RESULTS

With respect to acute cough (< 3 weeks), only three studies met our criteria for quality assessment, and all had a high risk of bias. As predicted by the 2006 CHEST Cough Guidelines, the most common causes were respiratory infections, most likely of viral cause, followed by exacerbations of underlying diseases such as asthma and COPD and pneumonia. The subjects resided on three continents: North America, Europe, and Asia. With respect to subacute cough (duration, 3-8 weeks), only two studies met our criteria for quality assessment, and both had a high risk of bias. As predicted by the 2006 guidelines, the most common causes were postinfectious cough and exacerbation of underlying diseases such as asthma, COPD, and upper airway cough syndrome (UACS). The subjects resided in countries in Asia. With respect to chronic cough (> 8 weeks), 11 studies met our criteria for quality assessment, and all had a high risk of bias. As predicted by the 2006 guidelines, the most common causes were UACS from rhinosinus conditions, asthma, gastroesophageal reflux disease, nonasthmatic eosinophilic bronchitis, combinations of these four conditions, and, less commonly, a variety of miscellaneous conditions and atopic cough in Asian countries. The subjects resided on four continents: North America, South America, Europe, and Asia.

CONCLUSIONS

Although the quality of evidence was low, the published literature since 2006 suggests that CHEST's 2006 Cough Guidelines and management algorithms for acute, subacute, and chronic cough in adults appeared useful in diagnosing and treating patients with cough around the globe. These same algorithms have been updated to reflect the advances in cough management as of 2017.

Cough in Ambulatory Immunocompromised Adults: CHEST Expert Panel Report

BACKGROUND

Cough is a common symptom prompting patients to seek medical care. Like patients in the general population, patients with compromised immune systems also seek care for cough. However, it is unclear whether the causes of cough in immunocompromised patients who are deemed unlikely to have a life-threating condition and a normal or unchanged chest radiograph are similar to those in persons with cough and normal immune systems.

METHODS

We conducted a systematic review to answer the question: What are the most common causes of cough in ambulatory immunodeficient adults with normal chest radiographs? Studies of patients ≥ 18 years of age with immune deficiency, cough of any duration, and normal or unchanged chest radiographs were included and assessed for relevance and quality. Based on the systematic review, suggestions were developed and voted on using the American College of Chest Physicians (CHEST) methodology framework.

RESULTS

The results of the systematic review revealed no high-quality evidence to guide the clinician in determining the likely causes of cough specifically in immunocompromised ambulatory patients with normal chest radiographs.

CONCLUSIONS

Based on a systematic review, we found no evidence to assess whether or not the proper initial evaluation of cough in immunocompromised patients is different from that in immunocompetent persons. A consensus of the panel suggested that the initial diagnostic algorithm should be similar to that for immunocompetent persons but that the context of the type and severity of the immune defect, geographic location, and social determinants be considered. The major modifications to the 2006 CHEST Cough Guidelines are the suggestions that TB should be part of the initial evaluation of patients with cough and HIV infection who reside in regions with a high prevalence of TB, regardless of the radiographic findings, and that specific causes and immune defects be considered in all patients in whom the initial evaluation is unrevealing.

Enhancing sensory experiences for very preterm infants in the NICU: an integrative review

OBJECTIVE

Very preterm infants hospitalized in the neonatal intensive care unit (NICU) experience alterations in sensory experiences. Defining types, timing and frequency of sensory-based interventions that optimize outcomes can inform environmental modifications. The objective of this study was to conduct an integrative review on sensory-based interventions used with very preterm infants in the NICU to improve infant and parent outcomes.

STUDY DESIGN

The data sources include MEDLINE, CINAHL, Cochrane Library and Google Scholar. Studies were identified that used sensory-based interventions in the NICU with preterm infants born ⩽32 weeks gestation, were published in a peer-reviewed journal between 1995 and 2015, and measured outcomes related to infant and parent outcomes. Studies were extracted from electronic databases and hand-searched from identified reference lists.

RESULTS

Eighty-eight articles were identified (31 tactile, 12 auditory, 3 visual, 2 kinesthetic, 2 gustatory/olfactory and 37 multimodal). There was evidence to support the use of kangaroo care, music and language exposure, and multimodal interventions starting at 25 to 28 weeks postmenstrual age. These interventions were related to better infant development and lower maternal stress, but not all findings were consistent. Limitations included lack of consistent outcome measures, study quality and gaps in the literature.

CONCLUSIONS

Most research identified interventions that were done for short periods of time. It is unclear what the potential is for improving outcomes if positive sensory exposures occur consistently throughout NICU hospitalization. Until more research defines appropriate sensory-based interventions to use with infants born very preterm in the NICU, information from this review can be combined with expert opinion and parent/family values to determine best practice.