Discordances originated by multiple meta-analyses on interventions for myocardial infarction: a systematic review

Exploring decision-makers' challenges and strategies when selecting multiple systematic reviews: insights for AI decision support tools in healthcare

BACKGROUND

Systematic reviews (SRs) are being published at an accelerated rate. Decision-makers may struggle with comparing and choosing between multiple SRs on the same topic. We aimed to understand how healthcare decision-makers (eg, practitioners, policymakers, researchers) use SRs to inform decision-making and to explore the potential role of a proposed artificial intelligence (AI) tool to assist in critical appraisal and choosing among SRs.

METHODS

We developed a survey with 21 open and closed questions. We followed a knowledge translation plan to disseminate the survey through social media and professional networks.

RESULTS

Our survey response rate was lower than expected (7.9% of distributed emails). Of the 684 respondents, 58.2% identified as researchers, 37.1% as practitioners, 19.2% as students and 13.5% as policymakers. Respondents frequently sought out SRs (97.1%) as a source of evidence to inform decision-making. They frequently (97.9%) found more than one SR on a given topic of interest to them. Just over half (50.8%) struggled to choose the most trustworthy SR among multiple. These difficulties related to lack of time (55.2%), or difficulties comparing due to varying methodological quality of SRs (54.2%), differences in results and conclusions (49.7%) or variation in the included studies (44.6%). Respondents compared SRs based on the relevance to their question of interest, methodological quality, and recency of the SR search. Most respondents (87.0%) were interested in an AI tool to help appraise and compare SRs.

CONCLUSIONS

Given the identified barriers of using SR evidence, an AI tool to facilitate comparison of the relevance of SRs, the search and methodological quality, could help users efficiently choose among SRs and make healthcare decisions.

Wearable Devices to Improve Physical Activity and Reduce Sedentary Behaviour: An Umbrella Review

BACKGROUND

Several systematic reviews (SRs), with and without meta-analyses, have investigated the use of wearable devices to improve physical activity, and there is a need for frequent and updated syntheses on the topic.

OBJECTIVE

We aimed to evaluate whether using wearable devices increased physical activity and reduced sedentary behaviour in adults.

METHODS

We conducted an umbrella review searching PubMed, Cumulative Index to Nursing and Allied Health Literature, the Cochrane Library, MedRxiv, Rxiv and bioRxiv databases up to February 5th, 2023. We included all SRs that evaluated the efficacy of interventions when wearable devices were used to measure physical activity in adults aged over 18 years. The primary outcomes were physical activity and sedentary behaviour measured as the number of steps per day, minutes of moderate to vigorous physical activity (MVPA) per week, and minutes of sedentary behaviour (SB) per day. We assessed the methodological quality of each SR using the Assessment of Multiple Systematic Reviews, version 2 (AMSTAR 2) and the certainty of evidence of each outcome measure using the GRADE (Grading of Recommendations, Assessment, Development, and Evaluations). We interpreted the results using a decision-making framework examining the clinical relevance and the concordances or discordances of the SR effect size.

RESULTS

Fifty-one SRs were included, of which 38 included meta-analyses (302 unique primary studies). Of the included SRs, 72.5% were rated as 'critically low methodological quality'. Overall, with a slight overlap of primary studies (corrected cover area: 3.87% for steps per day, 3.12% for MVPA, 4.06% for SB) and low-to-moderate certainty of the evidence, the use of WDs may increase PA by a median of 1,312.23 (IQR 627-1854) steps per day and 57.8 (IQR 37.7 to 107.3) minutes per week of MVPA. Uncertainty is present for PA in pathologies and older adults subgroups and for SB in mixed and older adults subgroups (large confidence intervals).

CONCLUSIONS

Our findings suggest that the use of WDs may increase physical activity in middle-aged adults. Further studies are needed to investigate the effects of using WDs on specific subgroups (such as pathologies and older adults) in different follow-up lengths, and the role of other intervention components.

Effectiveness and safety of virtual reality rehabilitation after stroke: an overview of systematic reviews

Background

Virtual reality (VR) is an innovative neurorehabilitation modality that has been variously examined in systematic reviews. We assessed VR effectiveness and safety after cerebral stroke.

Methods

In this overview of systematic reviews, we searched eleven databases (Cochrane Database of Systematic Reviews, EMBASE, MEDLINE, SCOPUS, ISI Web of Science, CINAHL, PsycINFO, Pedro, Otseeker, Healthevidence.org, Epistemonikos) and grey literature from inception to January 17, 2023. Studies eligible for inclusion were systematic reviews published in English that included adult patients with a clinical diagnosis of stroke (acute to chronic phase) undergoing any kind of immersive, semi-immersive or non-immersive VR intervention with or without conventional therapy versus conventional therapy alone. The primary outcome was motor upper limb function and activity. The secondary outcomes were gait and balance, cognitive and mental function, limitation of activities, participation, and adverse events. We calculated the degree of overlap between reviews based on the corrected covered area (CCA). Methodological quality was assessed using the A MeaSurement Tool to Assess systematic Reviews (AMSTAR 2) and the Certainty of Evidence (CoE) using the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) approach. Discordances between results were examined using a conceptual framework based on the Jadad algorithm. This overview is registered with PROSPERO, CRD42022329263.

Findings

Of the 58 reviews included (n = 345 unique primary studies), 42 (72.4%) had conducted meta-analysis. More than half of the reviews (58.6%) were published between 2020 and 2022 and many (77.6%) were judged critically low in quality by AMSTAR 2. Most reported the Fugl Meyer Assessment scale (FMA-UE) to measure upper limb function and activity. For the primary outcome, there was a moderate overlap of primary studies (CCA 9.0%) with discordant findings. Focusing on upper limb function (FMA-UE), VR with or without conventional therapy seems to be more effective than conventional therapy alone, with low to moderate CoE and probable to definite clinical relevance. For secondary outcomes there was uncertainty about the superiority or no difference between groups due to substantial heterogeneity of measurement scales (eg, methodological choices). A few reviews (n = 6) reported the occurrence of mild adverse events.

Interpretation

Current evidence suggests that multiple meta-analyses agreed on the superiority of VR with or without conventional therapy over conventional therapy on FME-UE for upper limb. Clinicians may consider embedding VR technologies into their practice as appropriate with patient's goals, abilities, and preferences. However, caution is needed given the poor methodological quality of reviews.

Funding

Italian Ministry of Health.

Structured approach with primary and secondary survey for major trauma care: an overview of reviews

BACKGROUND

A structured approach involves systematic management of trauma patients. We aim to conduct an overview of reviews about the clinical efficacy and safety of structured approach (i.e., primary and secondary survey) by guideline checklist compared to non-structured approach (i.e. clinical examination); moreover, routine screening whole-body computer tomography (WBCT) was compared to non-routine WBCT in patients with suspected major trauma.

METHODS

We systematically searched MEDLINE (PubMed), EMBASE and Cochrane Database of Systematic Reviews up to 3 May 2022. Systematic reviews (SRs) that investigated the use of a structured approach compared to a non-structured approach were eligible. Two authors independently extracted data, managed the overlapping of primary studies belonging to the included SRs and calculated the corrected covered area (CCA). The certainty of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology.

RESULTS

We included nine SRs investigating two comparisons in stable trauma patients: structured approach vs non-structured approach (n = 1) and routine WBCT vs non-routine WBCT (n = 8). The overlap of included primary studies was generally high across outcomes (CCA ranged between 20.85 and 42.86%) with some discrepancies in the directions of effects across reviews. The application of a structured approach by checklist may improve adherence to guidelines (e.g. Advanced Trauma Life Support) during resuscitation and might lead to a reduction in mortality among severely injured patients as compared to clinical examination (Adjusted OR 0.51; 95% CI 0.30-0.89; p = 0.018; low certainty of evidence). The use of routine WBCT seems to offer little to no effects in reducing mortality and time spent in emergency room or department, whereas non-routine WBCT seems to offer little to no effects in reducing radiation dose, intensive care unit length of stay (LOS) and hospital LOS (low-to-moderate certainty of evidence).

CONCLUSIONS

The application of structured approach by checklist during trauma resuscitation may improve patient- and process-related outcomes. Including non-routine WBCT seems to offer the best trade-offs between benefits and harm. Clinicians should consider these findings in the light of their clinical context, the volume of patients in their facilities, the need for time management, and costs.

Ocular Biomarkers for Alzheimer Disease Dementia: An Umbrella Review of Systematic Reviews and Meta-analyses

Importance

Several ocular biomarkers have been proposed for the early detection of Alzheimer disease (AD) and mild cognitive impairment (MCI), particularly fundus photography, optical coherence tomography (OCT), and OCT angiography (OCTA).

Objective

To perform an umbrella review of systematic reviews to assess the diagnostic accuracy of ocular biomarkers for early diagnosis of Alzheimer disease.

Data Sources

MEDLINE, Embase, and PsycINFO were searched from January 2000 to November 2021. The references of included reviews were also searched.

Study Selection

Systematic reviews investigating the diagnostic accuracy of ocular biomarkers to detect AD and MCI, in secondary care or memory clinics, against established clinical criteria or clinical judgment.

Data Extraction and Synthesis

The Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline checklist was followed and the Risk Of Bias in Systematic reviews tool was used to assess review quality.

Main Outcomes and Measures

The prespecified outcome was the accuracy of ocular biomarkers for diagnosing AD and MCI. The area under the curve (AUC) was derived from standardized mean difference.

Results

From the 591 titles, 14 systematic reviews were included (median [range] number of studies in each review, 14 [5-126]). Only 4 reviews were at low risk of bias on all Risk of Bias in Systematic Reviews domains. The imaging-derived parameters with the most evidence for detecting AD compared with healthy controls were OCT peripapillary retinal nerve fiber layer thickness (38 studies including 1883 patients with AD and 2510 controls; AUC = 0.70; 95% CI, 0.53-0.79); OCTA foveal avascular zone (5 studies including 177 patients with AD and 371 controls; AUC = 0.73; 95% CI, 0.50-0.89); and saccadic eye movements prosaccade latency (30 studies including 651 patients with AD/MCI and 771 controls; AUC = 0.64; 95% CI, 0.58-0.69). Antisaccade error was investigated in fewer studies (12 studies including 424 patients with AD/MCI and 382 controls) and yielded the best accuracy (AUC = 0.79; 95% CI, 0.70-0.88).

Conclusions and Relevance

This umbrella review has highlighted limitations in design and reporting of the existing research on ocular biomarkers for diagnosing AD. Parameters with the best evidence showed poor to moderate diagnostic accuracy in cross-sectional studies. Future longitudinal studies should investigate whether changes in OCT and OCTA measurements over time can yield accurate predictions of AD onset.

A new taxonomy was developed for overlap across 'overviews of systematic reviews': A meta-research study of research waste

Multiple 'overviews of reviews' conducted on the same topic ("overlapping overviews") represent a waste of research resources and can confuse clinicians making decisions amongst competing treatments. We aimed to assess the frequency and characteristics of overlapping overviews. MEDLINE, Epistemonikos and Cochrane Database of Systematic Reviews were searched for overviews that: synthesized reviews of health interventions and conducted systematic searches. Overlap was defined as: duplication of PICO eligibility criteria, and not reported as an update nor a replication. We categorized overview topics according to 22 WHO ICD-10 medical classifications, overviews as broad or narrow in scope, and overlap as identical, nearly identical, partial, or subsumed. Subsummation was defined as when broad overviews subsumed the populations, interventions and at least one outcome of another overview. Of 541 overviews included, 169 (31%) overlapped across similar PICO, fell within 13 WHO ICD-10 medical classifications, and 62 topics. 148/169 (88%) overlapping overviews were broad in scope. Fifteen overviews were classified as having nearly identical overlap (9%); 123 partial overlap (73%), and 31 subsumed (18%) others. One third of overviews overlapped in content and a majority covered broad topic areas. A multiplicity of overviews on the same topic adds to the ongoing waste of research resources, time, and effort across medical disciplines. Authors of overviews can use this study and the sample of overviews to identify gaps in the evidence for future analysis, and topics that are already studied, which do not need to be duplicated.

Mitigating Disputes Originated by Multiple Discordant Systematic Reviews and Meta-Analyses: A Survey of Methodologists and Clinicians

Background

Overlapping systematic reviews (SRs) are increasingly frequent in the medical literature. They can easily generate discordant evidence, as estimates of effect sizes and their interpretation might differ from one source to another.

Objective

To analyze how methodologists and clinicians make a decision when faced with discordant evidence formalized in structured tables.

Methods

We conducted a 16-item survey exploring how methodologists and clinicians would react when presented with multiple Summary of Findings (SoF) tables (generated using the GRADE tool) derived from 4 overlapping and discordant SRs and meta-analyses on thrombolytic therapy for intermediate-risk pulmonary embolism. SoF tables reported 4 different magnitudes of effects and overall certainty. Participants were asked to provide their recommendations regarding the intervention and the reasons behind their conclusion.

Results

Of the 80 invitees, 41 (51%) participated. The majority described themselves as "somewhat familiar" or experts with SoF tables. The majority recommended the therapy (pharmacological systemic thrombolysis), grading the recommendation as weak positive. Certainty of evidence and benefit-risk balance were the two criteria that prevailed in generating the recommendation. When faced with overlapping meta-analyses, the preferred approach was to use only high-quality SRs and exclude redundant SRs. Several participants suggested integrating the SoF tables with additional information, such as a more comprehensive evaluation of the risk of bias of systematic reviews (71%), heterogeneity/inconsistency (68%) and studies included within each SR (62%).

Conclusion

When faced with multiple controversial SR results, the type and completeness of reported information in SoF tables affect experts' ability to make recommendations. Developers of the SoF table should consider collating key information from overlapping and potentially discordant reviews.

Overlapping network meta-analyses on psoriasis systemic treatments: an overview, quantity does not make quality

BACKGROUND

Network Meta-analyses (NMAs) have become successful in addressing gaps in the comparative effectiveness of systemic treatments in moderate-to-severe psoriasis. However, their increasing number carries both a risk of overlap and reproducibility issues that can hamper clinical decision-making. In this overview, we aimed to assess redundancy across these NMAs and to describe their characteristics.

METHODS

We considered all systematic reviews with NMAs of randomized controlled trials that included adult patients with moderate-to-severe psoriasis and that evaluated the efficacy and/or safety of systemic treatments compared with placebo or with an active comparator. PubMed/MEDLINE, Epistemonikos, PROSPERO and the Evidence Update of the Centre of Evidence-Based Dermatology of the University of Nottingham were searched up to 25 February 2021. Our main outcome was the number per year of redundant NMAs and the extent of their overlap. We also described their features, especially, the confidence in the results of the reviews, the studies' funding and the presence of spin (a description that overstates efficacy and/or understates harm), reporting issues and methodological characteristics.

RESULTS

In total, 47 redundant NMAs were included. Only 2/47 (4%) included all available treatments. Both efficacy and safety were evaluated in 14/47 (30%) NMAs and both short and long-term evaluations were assessed in 5/47 (11 %). Confidence in the results was critically low for 39/47 (83%) NMAs and only 10/47 (23 %) registered a protocol. 26/47 NMAs (55%) received pharmaceutical funding. CROs were involved in 19/47 (40%) NMAs. Reporting was poor across most of the NMAs' abstracts and spin was present in all of the abstracts. Almost half of the NMA failed to consider important limitations such as heterogeneity (32%) or consistency (66%).

DISCUSSION

In addition to a duplication of efforts, our overview showed heterogeneous methods and poor confidence in the results in a majority of the included NMAs, further distorted by reporting issues and spin. Clinicians need to interpret NMAs with caution when looking for the most reliable and comprehensive evidence.

Accuracy of optical coherence tomography for diagnosing glaucoma: an overview of systematic reviews

AIMS

To assess the diagnostic accuracy (DTA) of optical coherence tomography (OCT) for detecting glaucoma by systematically searching and appraising systematic reviews (SRs) on this issue.

METHODS

We searched a database of SRs in eyes and vision maintained by the Cochrane Eyes and Vision United States on the DTA of OCT for detecting glaucoma. Two authors working independently screened the records, abstracted data and assessed the risk of bias using the Risk of Bias in Systematic Reviews checklist. We extracted quantitative DTA estimates as well as qualitative statements on their relevance to practice.

RESULTS

We included four SRs published between 2015 and 2018. These SRs included between 17 and 113 studies on OCT for glaucoma diagnosis. Two reviews were at low risk of bias and the other two had two to four domains at high or unclear risk of bias with concerns on applicability. The two reliable SRs reported the accuracy of average retinal nerve fibre layer (RNFL) thickness and found a sensitivity of 0.69 (0.63 to 0.73) and 0.78 (0.74 to 0.83) and a specificity of 0.94 (0.93 to 0.95) and 0.93 (0.92 to 0.95) in 57 and 50 studies, respectively. Only one review included a clear specification of the clinical pathway. Both reviews highlighted the limitations of primary DTA studies on this topic.

CONCLUSIONS

The quality of published DTA reviews on OCT for diagnosing glaucoma was mixed. Two reliable SRs found moderate sensitivity at high specificity for average RNFL thickness in diagnosing manifest glaucoma. Our overview suggests that the methodological quality of both primary and secondary DTA research on glaucoma is in need of improvement.

Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis

BACKGROUND

Different methodological choices such as inclusion/exclusion criteria and analytical models can yield different results and inferences when meta-analyses are performed. We explored the range of such differences, using several methodological choices for indirect comparison meta-analyses to compare nalmefene and naltrexone in the reduction of alcohol consumption as a case study.

METHODS

All double-blind randomized controlled trials (RCTs) comparing nalmefene to naltrexone or one of these compounds to a placebo in the treatment of alcohol dependence or alcohol use disorders were considered. Two reviewers searched for published and unpublished studies in MEDLINE (August 2017), the Cochrane Library, Embase, and ClinicalTrials.gov and contacted pharmaceutical companies, the European Medicines Agency, and the Food and Drug Administration. The indirect comparison meta-analyses were performed according to different inclusion/exclusion criteria (based on medical condition, abstinence of patients before inclusion, gender, somatic and psychiatric comorbidity, psychological support, treatment administered and dose, treatment duration, outcome reported, publication status, and risk of bias) and different analytical models (fixed and random effects). The primary outcome was the vibration of effects (VoE), i.e. the range of different results of the indirect comparison between nalmefene and naltrexone. The presence of a "Janus effect" was investigated, i.e. whether the 1st and 99th percentiles in the distribution of effect sizes were in opposite directions.

RESULTS

Nine nalmefene and 51 naltrexone RCTs were included. No study provided a direct comparison between the drugs. We performed 9216 meta-analyses for the indirect comparison with a median of 16 RCTs (interquartile range = 12-21) included in each meta-analysis. The standardized effect size was negative at the 1st percentile (- 0.29, favouring nalmefene) and positive at the 99th percentile (0.29, favouring naltrexone). A total of 7.1% (425/5961) of the meta-analyses with a negative effect size and 18.9% (616/3255) of those with a positive effect size were statistically significant (p < 0.05).

CONCLUSIONS

The choice of inclusion/exclusion criteria and analytical models for meta-analysis can result in entirely opposite results. VoE evaluations could be performed when overlapping meta-analyses on the same topic yield contradictory result.

TRIAL REGISTRATION

This study was registered on October 19, 2016, in the Open Science Framework (OSF, protocol available at https://osf.io/7bq4y/ ).

Interventions for Age-Related Macular Degeneration: Are Practice Guidelines Based on Systematic Reviews?

PURPOSE

Are existing systematic reviews of interventions for age-related macular degeneration incorporated into clinical practice guidelines?

DESIGN

High-quality systematic reviews should be used to underpin evidence-based clinical practice guidelines and clinical care. We examined the reliability of systematic reviews of interventions for age-related macular degeneration (AMD) and described the main findings of reliable reviews in relation to clinical practice guidelines.

METHODS

Eligible publications were systematic reviews of the effectiveness of treatment interventions for AMD. We searched a database of systematic reviews in eyes and vision without language or date restrictions; the database was up to date as of May 6, 2014. Two authors independently screened records for eligibility and abstracted and assessed the characteristics and methods of each review. We classified reviews as reliable when they reported eligibility criteria, comprehensive searches, methodologic quality of included studies, appropriate statistical methods for meta-analysis, and conclusions based on results. We mapped treatment recommendations from the American Academy of Ophthalmology (AAO) Preferred Practice Patterns (PPPs) for AMD to systematic reviews and citations of reliable systematic reviews to support each treatment recommendation.

RESULTS

Of 1570 systematic reviews in our database, 47 met inclusion criteria; most targeted neovascular AMD and investigated anti-vascular endothelial growth factor (VEGF) interventions, dietary supplements, or photodynamic therapy. We classified 33 (70%) reviews as reliable. The quality of reporting varied, with criteria for reliable reporting met more often by Cochrane reviews and reviews whose authors disclosed conflicts of interest. Anti-VEGF agents and photodynamic therapy were the only interventions identified as effective by reliable reviews. Of 35 treatment recommendations extracted from the PPPs, 15 could have been supported with reliable systematic reviews; however, only 1 recommendation cited a reliable intervention systematic review. No reliable systematic review was identified for 20 treatment recommendations, highlighting areas of evidence gaps.

CONCLUSIONS

For AMD, reliable systematic reviews exist for many treatment recommendations in the AAO PPPs and should be cited to support these recommendations. We also identified areas where no high-level evidence exists. Mapping clinical practice guidelines to existing systematic reviews is one way to highlight areas where evidence generation or evidence synthesis is either available or needed.