1
|
Guelimi R, Afach S, Bettuzzi T, Meyer A, Padern G, Yiu Z, Naudet F, Sbidian E, Le-Cleach L. Funding and conclusions of network meta-analyses on targeted therapies in inflammatory diseases: an overview. J Clin Epidemiol 2024; 172:111411. [PMID: 38852893 DOI: 10.1016/j.jclinepi.2024.111411] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2024] [Revised: 05/08/2024] [Accepted: 06/03/2024] [Indexed: 06/11/2024]
Abstract
OBJECTIVES To explore the association between industry funding and network meta-analyses' (NMAs) conclusion, and the use in Clinical Practice Guidelines (CPGs) of NMAs. STUDY DESIGN AND SETTING This was an overview of NMAs and CPGs. We searched PubMed/MEDLINE, Epistemonikos, and several guideline databases up to February 18th 2023. We included CPGs from the last 5 years and NMAs of randomized controlled trials that evaluated targeted therapies in immune-mediated inflammatory diseases. Data extraction and outcome assessments were done in duplicate by independent authors. RESULTS We included 216 NMAs and 99 CPGs. 31% (67/216) were industry-funded. The proportion of industry-funded NMAs that cited one treatment as being best was 44% (25/57) compared to 26% (30/116) for nonindustry-funded (OR = 2.24 [1.15-4.39]; aOR = 1.76 [0.81-3.81]). The abstract's conclusion of 39/67 (58%) industry-funded and 69/149 (46%) nonindustry-funded NMAs were considered unsupported by the results (OR = 1.61 [0.90-2.89]; aOR = 1.40 [0.71-2.78]). All industry-funded NMAs that cited one treatment as best cited their own sponsored drug. 59/99 (60%) CPGs included at least one NMA, with 23/59 (39%) of them citing industry-funded NMAs. CONCLUSIONS We did not find evidence that industry-funded NMAs were more likely to have unsupported conclusions or to cite only one treatment as being best in their conclusions compared to non-industry-funded NMAs. However, almost all industry-funded NMAs favored their own treatments. Even though 40% of the CPGs did not rely on NMA, over a third of those who did used industry-funded NMAs. Limitations include the possible misclassification due to undisclosed funding and potential confounders that have not been accounted for.
Collapse
Affiliation(s)
- Robin Guelimi
- EpiDermE EA 7379, Université Paris Est Créteil, Créteil, F-94010, France; Department of Dermatology, Assistance Publique-Hôpitaux de Paris, Henri Mondor Hospital, Créteil, F-94010, France.
| | - Sivem Afach
- EpiDermE EA 7379, Université Paris Est Créteil, Créteil, F-94010, France
| | - Thomas Bettuzzi
- EpiDermE EA 7379, Université Paris Est Créteil, Créteil, F-94010, France; Department of Dermatology, Assistance Publique-Hôpitaux de Paris, Henri Mondor Hospital, Créteil, F-94010, France
| | - Antoine Meyer
- Department of Gastroenterology, University Hospital of Bicêtre, Assistance Publique-Hôpitaux de Paris and Université Paris-Saclay, Le Kremlin Bicêtre, France
| | - Guillaume Padern
- IRMB, University of Montpellier, Inserm U1183, CHU Montpellier, Montpellier, France
| | - Zenas Yiu
- Faculty of Biology, Medicine and Health, Division of Musculoskeletal and Dermatological Sciences, School of Biological Sciences, The University of Manchester, Manchester, United Kingdom; Salford Royal Hospital, Northern Care Alliance NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, M6 8HD, United Kingdom
| | - Florian Naudet
- Univ Rennes, CHU Rennes, Inserm, Irset (Institut de Recherche en Santé, Environnement et Travail) - UMR_S 1085, CIC 1414 [(Centre d'Investigation Clinique de Rennes)], F-35000, Rennes, France; Institut Universitaire de France (IUF), Paris, France
| | - Emilie Sbidian
- EpiDermE EA 7379, Université Paris Est Créteil, Créteil, F-94010, France; Department of Dermatology, Assistance Publique-Hôpitaux de Paris, Henri Mondor Hospital, Créteil, F-94010, France
| | - Laurence Le-Cleach
- EpiDermE EA 7379, Université Paris Est Créteil, Créteil, F-94010, France; Department of Dermatology, Assistance Publique-Hôpitaux de Paris, Henri Mondor Hospital, Créteil, F-94010, France
| |
Collapse
|
2
|
Kolaski K, Logan LR, Ioannidis JPA. Improving systematic reviews: guidance on guidance and other options and challenges. J Clin Epidemiol 2023; 159:266-273. [PMID: 37196861 DOI: 10.1016/j.jclinepi.2023.05.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2023] [Revised: 05/07/2023] [Accepted: 05/09/2023] [Indexed: 05/19/2023]
Affiliation(s)
- Kat Kolaski
- Department of Orthopaedic Surgery, Wake Forest School of Medicine, Winston-Salem, NC, USA; Department of Pediatrics, Wake Forest School of Medicine, Winston-Salem, NC, USA; Department of Neurology, Wake Forest School of Medicine, Winston-Salem, NC, USA
| | - Lynne Romeiser Logan
- Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, NY, USA
| | - John P A Ioannidis
- Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA; Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, USA; Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA; Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA, USA; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA.
| |
Collapse
|
3
|
de Sousa PG, Mainka FF, Tonin FS, Pontarolo R. Mapping the characteristics, methodological quality and standards of reporting of network meta-analyses on antithrombotic therapies: An overview. Int J Cardiol 2023:S0167-5273(23)00729-5. [PMID: 37230428 DOI: 10.1016/j.ijcard.2023.05.036] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Revised: 04/20/2023] [Accepted: 05/19/2023] [Indexed: 05/27/2023]
Abstract
BACKGROUND Although a large number of network meta-analyses (NMAs) in the field of cardiology are available, little is known about their methodological quality. We aimed to map the characteristics and critically appraised the standards of conduct and evidence reporting of NMAs assessing antithrombotic therapies for the treatment or prophylaxis of heart diseases and cardiac surgical procedures. METHODS We systematically searched PubMed and Scopus to identify NMAs comparing the clinical effects of antithrombotic therapies. Overall characteristics of the NMAs were extracted and their reporting quality and methodological quality were evaluated using the PRISMA-NMA checklist and AMSTAR-2, respectively. RESULTS We found 86 NMAs published between 2007 and 2022. Comparisons among direct-acting oral anticoagulants were available in 61 (71%) NMAs. Although around 75% of NMAs stated that they followed international guidelines for conduct and reporting, only one third provided a protocol/register. Complete search strategies and publication bias assessment were lacking in around 53% and 59% of studies, respectively. Most NMAs (n = 77, 90%) provided supplemental material; however, only 5 (6%) made the complete raw data available. Network diagrams were depicted in most studies (n = 67, 78%), yet network geometry was described in only 11 (12.8%) of them. Mean adherence to the PRISMA-NMA checklist was 65.1 ± 16.5%. AMSTAR-2 assessment showed 88% of the NMAs had critically low methodological quality. CONCLUSION Although there is a wide diffusion of NMA-type studies on antithrombotics for heart diseases, their methodological and reporting quality remains suboptimal. This may reflect fragile clinical practices due to misleading conclusions from critically low-quality NMAs.
Collapse
Affiliation(s)
- Patricia Guerrero de Sousa
- Department of Medical and Pharmaceutical Sciences, Universidade Estadual do Oeste do Paraná, Cascavel, Brazil.
| | - Felipe Fernando Mainka
- Pharmaceutical Sciences Postgraduate Programme, Universidade Federal do Paraná, Curitiba, Brazil
| | - Fernanda Stumpf Tonin
- Pharmaceutical Sciences Postgraduate Programme, Universidade Federal do Paraná, Curitiba, Brazil; Health & Technology Research Centre, Escola Superior de Tecnologia da Saúde (H&TRC-ESTeSL), Instituto Politécnico de Lisboa, Lisbon, Portugal.
| | - Roberto Pontarolo
- Department of Pharmacy, Universidade Federal do Paraná, Curitiba, Brazil.
| |
Collapse
|
4
|
Guelimi R, Afach S, Régnaux JP, Bettuzzi T, Chaby G, Sbidian E, Naudet F, Le Cleach L. Overlapping network meta-analyses on psoriasis systemic treatments: an overview, quantity does not make quality. Br J Dermatol 2021; 187:29-41. [PMID: 34854074 DOI: 10.1111/bjd.20908] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2021] [Revised: 11/27/2021] [Accepted: 11/28/2021] [Indexed: 11/29/2022]
Abstract
BACKGROUND Network Meta-analyses (NMAs) have become successful in addressing gaps in the comparative effectiveness of systemic treatments in moderate-to-severe psoriasis. However, their increasing number carries both a risk of overlap and reproducibility issues that can hamper clinical decision-making. In this overview, we aimed to assess redundancy across these NMAs and to describe their characteristics. METHODS We considered all systematic reviews with NMAs of randomized controlled trials that included adult patients with moderate-to-severe psoriasis and that evaluated the efficacy and/or safety of systemic treatments compared with placebo or with an active comparator. PubMed/MEDLINE, Epistemonikos, PROSPERO and the Evidence Update of the Centre of Evidence-Based Dermatology of the University of Nottingham were searched up to 25 February 2021. Our main outcome was the number per year of redundant NMAs and the extent of their overlap. We also described their features, especially, the confidence in the results of the reviews, the studies' funding and the presence of spin (a description that overstates efficacy and/or understates harm), reporting issues and methodological characteristics. RESULTS In total, 47 redundant NMAs were included. Only 2/47 (4%) included all available treatments. Both efficacy and safety were evaluated in 14/47 (30%) NMAs and both short and long-term evaluations were assessed in 5/47 (11 %). Confidence in the results was critically low for 39/47 (83%) NMAs and only 10/47 (23 %) registered a protocol. 26/47 NMAs (55%) received pharmaceutical funding. CROs were involved in 19/47 (40%) NMAs. Reporting was poor across most of the NMAs' abstracts and spin was present in all of the abstracts. Almost half of the NMA failed to consider important limitations such as heterogeneity (32%) or consistency (66%). DISCUSSION In addition to a duplication of efforts, our overview showed heterogeneous methods and poor confidence in the results in a majority of the included NMAs, further distorted by reporting issues and spin. Clinicians need to interpret NMAs with caution when looking for the most reliable and comprehensive evidence.
Collapse
Affiliation(s)
- R Guelimi
- EpiDermE EA 7379, Université Paris Est Créteil, F-94010, Créteil, France
| | - S Afach
- EpiDermE EA 7379, Université Paris Est Créteil, F-94010, Créteil, France
| | - J-P Régnaux
- EpiDermE EA 7379, Université Paris Est Créteil, F-94010, Créteil, France.,Ecole des Hautes Etudes en Santé Publique (EHESP), F-35000, Rennes, France
| | - T Bettuzzi
- EpiDermE EA 7379, Université Paris Est Créteil, F-94010, Créteil, France.,Department of Dermatology, Assistance Publique-Hôpitaux de Paris, Henri Mondor Hospital, F-94010, Créteil, France
| | - G Chaby
- Dermatology Department, Amiens-Picardie University Hospital Center, Amiens, F-80000, France
| | - E Sbidian
- EpiDermE EA 7379, Université Paris Est Créteil, F-94010, Créteil, France.,Department of Dermatology, Assistance Publique-Hôpitaux de Paris, Henri Mondor Hospital, F-94010, Créteil, France
| | - F Naudet
- CHU Rennes, INSERM CIC 1414 (Centre d'Investigation Clinique de Rennes), University Rennes, Rennes, F-35000, France
| | - L Le Cleach
- EpiDermE EA 7379, Université Paris Est Créteil, F-94010, Créteil, France.,Department of Dermatology, Assistance Publique-Hôpitaux de Paris, Henri Mondor Hospital, F-94010, Créteil, France
| |
Collapse
|
5
|
Putman MS, Chaitoff A, Niforatos JD. Publication Trends in Rheumatology Systematic Reviews and Randomized Clinical Trials, 1995-2017. J Rheumatol 2020; 48:305-306. [PMID: 33060312 DOI: 10.3899/jrheum.200593] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/11/2023]
Affiliation(s)
- Michael S Putman
- Northwestern Medicine, Department of Medicine, Division of Rheumatology, Chicago, Illinois;
| | - Alexander Chaitoff
- Department of Internal Medicine, Brigham and Women's Hospital/Harvard Medical School, Boston, Massachusetts
| | - Joshua D Niforatos
- Department of Emergency Medicine, The Johns Hopkins School of Medicine, Baltimore, Maryland, USA
| |
Collapse
|
6
|
In network meta-analysis, most of the information comes from indirect evidence: empirical study. J Clin Epidemiol 2020; 124:42-49. [DOI: 10.1016/j.jclinepi.2020.04.009] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2019] [Revised: 03/23/2020] [Accepted: 04/09/2020] [Indexed: 11/24/2022]
|
7
|
Boutron I, Créquit P, Williams H, Meerpohl J, Craig JC, Ravaud P. Future of evidence ecosystem series: 1. Introduction Evidence synthesis ecosystem needs dramatic change. J Clin Epidemiol 2020; 123:135-142. [DOI: 10.1016/j.jclinepi.2020.01.024] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2019] [Revised: 12/23/2019] [Accepted: 01/07/2020] [Indexed: 12/13/2022]
|
8
|
Braillon A, Naudet F. Meta-Analyses and the Janus Effect. PSYCHOTHERAPY AND PSYCHOSOMATICS 2020; 89:50. [PMID: 31715607 DOI: 10.1159/000504201] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/15/2019] [Accepted: 10/17/2019] [Indexed: 01/16/2023]
|
9
|
Palpacuer C, Hammas K, Duprez R, Laviolle B, Ioannidis JPA, Naudet F. Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis. BMC Med 2019; 17:174. [PMID: 31526369 PMCID: PMC6747755 DOI: 10.1186/s12916-019-1409-3] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/10/2019] [Accepted: 08/14/2019] [Indexed: 11/17/2022] Open
Abstract
BACKGROUND Different methodological choices such as inclusion/exclusion criteria and analytical models can yield different results and inferences when meta-analyses are performed. We explored the range of such differences, using several methodological choices for indirect comparison meta-analyses to compare nalmefene and naltrexone in the reduction of alcohol consumption as a case study. METHODS All double-blind randomized controlled trials (RCTs) comparing nalmefene to naltrexone or one of these compounds to a placebo in the treatment of alcohol dependence or alcohol use disorders were considered. Two reviewers searched for published and unpublished studies in MEDLINE (August 2017), the Cochrane Library, Embase, and ClinicalTrials.gov and contacted pharmaceutical companies, the European Medicines Agency, and the Food and Drug Administration. The indirect comparison meta-analyses were performed according to different inclusion/exclusion criteria (based on medical condition, abstinence of patients before inclusion, gender, somatic and psychiatric comorbidity, psychological support, treatment administered and dose, treatment duration, outcome reported, publication status, and risk of bias) and different analytical models (fixed and random effects). The primary outcome was the vibration of effects (VoE), i.e. the range of different results of the indirect comparison between nalmefene and naltrexone. The presence of a "Janus effect" was investigated, i.e. whether the 1st and 99th percentiles in the distribution of effect sizes were in opposite directions. RESULTS Nine nalmefene and 51 naltrexone RCTs were included. No study provided a direct comparison between the drugs. We performed 9216 meta-analyses for the indirect comparison with a median of 16 RCTs (interquartile range = 12-21) included in each meta-analysis. The standardized effect size was negative at the 1st percentile (- 0.29, favouring nalmefene) and positive at the 99th percentile (0.29, favouring naltrexone). A total of 7.1% (425/5961) of the meta-analyses with a negative effect size and 18.9% (616/3255) of those with a positive effect size were statistically significant (p < 0.05). CONCLUSIONS The choice of inclusion/exclusion criteria and analytical models for meta-analysis can result in entirely opposite results. VoE evaluations could be performed when overlapping meta-analyses on the same topic yield contradictory result. TRIAL REGISTRATION This study was registered on October 19, 2016, in the Open Science Framework (OSF, protocol available at https://osf.io/7bq4y/ ).
Collapse
Affiliation(s)
- Clément Palpacuer
- Centre d'Investigation Clinique INSERM 1414, Hôpital de Pontchaillou, 2 rue Henri le Guilloux, 35033, Rennes cedex 9, France. .,Department of Biostatistics, Institut de Cancérologie de l'Ouest Centre René-Gauducheau, Saint-Herblain, France.
| | - Karima Hammas
- Department of Epidemiology and Biostatistics and Clinical Research, Assistance Publique-Hôpitaux de Paris, Hôpital Bichat Claude Bernard, Paris, France.,Inserm, CIC-EC 1425, Hôpital Bichat Claude Bernard, Paris, France
| | - Renan Duprez
- Fondation Saint Jean de Dieu, Centre Hospitalier Dinan/St Brieuc, Dinan, France
| | - Bruno Laviolle
- Centre d'Investigation Clinique INSERM 1414, Hôpital de Pontchaillou, 2 rue Henri le Guilloux, 35033, Rennes cedex 9, France.,Department of Biological and Clinical Pharmacology and Pharmacovigilance, Rennes University Hospital, Rennes, France.,Laboratory of Experimental and Clinical Pharmacology, Rennes 1 University, Rennes, France
| | - John P A Ioannidis
- Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA.,Departments of Medicine, of Health Research and Policy, Biomedical Data Science, and Statistics, Stanford University, Stanford, CA, USA
| | - Florian Naudet
- Centre d'Investigation Clinique INSERM 1414, Hôpital de Pontchaillou, 2 rue Henri le Guilloux, 35033, Rennes cedex 9, France.,Department of Biological and Clinical Pharmacology and Pharmacovigilance, Rennes University Hospital, Rennes, France.,Laboratory of Experimental and Clinical Pharmacology, Rennes 1 University, Rennes, France.,Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA
| |
Collapse
|
10
|
Michelson M, Reuter K. The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials. Contemp Clin Trials Commun 2019; 16:100443. [PMID: 31497675 PMCID: PMC6722281 DOI: 10.1016/j.conctc.2019.100443] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2019] [Revised: 07/25/2019] [Accepted: 08/21/2019] [Indexed: 01/31/2023] Open
Abstract
Background More than 90% of clinical-trial compounds fail to demonstrate sufficient efficacy and safety. To help alleviate this issue, systematic literature review and meta-analysis (SLR), which synthesize current evidence for a research question, can be applied to preclinical evidence to identify the most promising therapeutics. However, these methods remain time-consuming and labor-intensive. Here, we introduce an economic formula to estimate the expense of SLR for academic institutions and pharmaceutical companies. Methods We estimate the manual effort involved in SLR by quantifying the amount of labor required and the total associated labor cost. We begin with an empirical estimation and derive a formula that quantifies and describes the cost. Results The formula estimated that each SLR costs approximately $141,194.80. We found that on average, the ten largest pharmaceutical companies publish 118.71 and the ten major academic institutions publish 132.16 SLRs per year. On average, the total cost of all SLRs per year to each academic institution amounts to $18,660,304.77 and for each pharmaceutical company is $16,761,234.71. Discussion It appears that SLR is an important, but costly mechanisms to assess the totality of evidence. Conclusions With the increase in the number of publications, the significant time and cost of SLR may pose a barrier to their consistent application to assess the promise of clinical trials thoroughly. We call on investigators and developers to develop automated solutions to help with the assessment of preclinical evidence particularly. The formula we introduce provides a cost baseline against which the efficiency of automation can be measured.
Collapse
Affiliation(s)
- Matthew Michelson
- Evid Science, 2361 Rosecrans Ave #348, El Segundo, 90245, Los Angeles, CA, United States
- Corresponding author.
| | - Katja Reuter
- Institute for Health Promotion and Disease Prevention Research, Department of Preventive Medicine, Keck School of Medicine, University of Southern California, 2001 N Soto St, Los Angeles, CA, 90032, United States
- Southern California Clinical and Translational Science Institute, Keck School of Medicine, University of Southern California, 2250 Alcazar, Los Angeles, CA, 90089, United States
| |
Collapse
|
11
|
Siontis KC, Ioannidis JPA. Replication, Duplication, and Waste in a Quarter Million Systematic Reviews and Meta-Analyses. Circ Cardiovasc Qual Outcomes 2019; 11:e005212. [PMID: 30562075 DOI: 10.1161/circoutcomes.118.005212] [Citation(s) in RCA: 41] [Impact Index Per Article: 8.2] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/22/2022]
Affiliation(s)
| | - John P A Ioannidis
- Stanford Prevention Research Center, Department of Medicine, Meta-Research Innovation Center at Stanford (METRICS). Departments of Health Research and Policy, Biomedical Data Science, and Statistics, Stanford University, CA (J.P.A.I.)
| |
Collapse
|
12
|
Hansen C, Lundh A, Rasmussen K, Hróbjartsson A. Financial conflicts of interest in systematic reviews: associations with results, conclusions, and methodological quality. Cochrane Database Syst Rev 2019; 8:MR000047. [PMID: 31425611 PMCID: PMC7040976 DOI: 10.1002/14651858.mr000047.pub2] [Citation(s) in RCA: 26] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/08/2023]
Abstract
BACKGROUND Financial conflicts of interest in systematic reviews (e.g. funding by drug or device companies or authors' collaboration with such companies) may impact on how the reviews are conducted and reported. OBJECTIVES To investigate the degree to which financial conflicts of interest related to drug and device companies are associated with results, conclusions, and methodological quality of systematic reviews. SEARCH METHODS We searched PubMed, Embase, and the Cochrane Methodology Register for studies published up to November 2016. We also read reference lists of included studies, searched grey literature sources, and Web of Science for studies citing the included studies. SELECTION CRITERIA Eligible studies were studies that compared systematic reviews with and without financial conflicts of interest in order to investigate differences in results (estimated treatment effect and frequency of statistically favourable results), frequency of favourable conclusions, or measures of methodological quality of the review (e.g. as evaluated on the Oxman and Guyatt index). DATA COLLECTION AND ANALYSIS Two review authors independently determined the eligibility of studies, extracted data, and assessed risk of bias. We synthesised the results of each study relevant to each of our outcomes. For meta-analyses, we used Mantel-Haenszel random-effects models to estimate risk ratios (RR) with 95% confidence intervals (CIs), with RR > 1 indicating that systematic reviews with financial conflicts of interest more frequently had statistically favourable results or favourable conclusions, and had lower methodological quality. When a quantitative synthesis was considered not meaningful, results from individual studies were summarised qualitatively. MAIN RESULTS Ten studies with a total of 995 systematic reviews of drug studies and 15 systematic reviews of device studies were included. We assessed two studies as low risk of bias and eight as high risk, primarily because of risk of confounding. The estimated treatment effect was not statistically significantly different for systematic reviews with and without financial conflicts of interest (Z-score: 0.46, P value: 0.64; based on one study of 14 systematic reviews which had a matched design, comparing otherwise similar systematic reviews). We found no statistically significant difference in frequency of statistically favourable results for systematic reviews with and without financial conflicts of interest (RR: 0.84, 95% CI: 0.62 to 1.14; based on one study of 124 systematic reviews). An analysis adjusting for confounding due to methodological quality (i.e. score on the Oxman and Guyatt index) provided a similar result. Systematic reviews with financial conflicts of interest more often had favourable conclusions compared with systematic reviews without (RR: 1.98, 95% CI: 1.26 to 3.11; based on seven studies of 411 systematic reviews). Similar results were found in two studies with a matched design, which therefore had a reduced risk of confounding. Systematic reviews with financial conflicts of interest tended to have lower methodological quality compared with systematic reviews without financial conflicts of interest (RR for 11 dimensions of methodological quality spanned from 1.00 to 1.83). Similar results were found in analyses based on two studies with matched designs. AUTHORS' CONCLUSIONS Systematic reviews with financial conflicts of interest more often have favourable conclusions and tend to have lower methodological quality than systematic reviews without financial conflicts of interest. However, it is uncertain whether financial conflicts of interest are associated with the results of systematic reviews. We suggest that patients, clinicians, developers of clinical guidelines, and planners of further research could primarily use systematic reviews without financial conflicts of interest. If only systematic reviews with financial conflicts of interest are available, we suggest that users read the review conclusions with skepticism, critically appraise the methods applied, and interpret the review results with caution.
Collapse
Affiliation(s)
- Camilla Hansen
- Odense University HospitalCentre for Evidence‐Based Medicine Odense (CEBMO)Kløvervænget 10, 13. FloorOdenseDenmark5000
- University of Southern DenmarkDepartment of Clinical ResearchOdenseDenmark
- Odense University HospitalOpen Patient data Explorative Network (OPEN)OdenseDenmark
- RigshospitaletNordic Cochrane CentreBlegdamsvej 9CopenhagenDenmark2200
| | - Andreas Lundh
- Odense University HospitalCentre for Evidence‐Based Medicine Odense (CEBMO)Kløvervænget 10, 13. FloorOdenseDenmark5000
- University of Southern DenmarkDepartment of Clinical ResearchOdenseDenmark
- Odense University HospitalOpen Patient data Explorative Network (OPEN)OdenseDenmark
- Hvidovre HospitalDepartment of Infectious DiseasesKettegård Allé 30HvidovreDenmark2650
| | | | - Asbjørn Hróbjartsson
- Odense University HospitalCentre for Evidence‐Based Medicine Odense (CEBMO)Kløvervænget 10, 13. FloorOdenseDenmark5000
- University of Southern DenmarkDepartment of Clinical ResearchOdenseDenmark
- Odense University HospitalOpen Patient data Explorative Network (OPEN)OdenseDenmark
| | | |
Collapse
|
13
|
Protect us from poor-quality medical research. Hum Reprod 2019; 33:770-776. [PMID: 29617882 DOI: 10.1093/humrep/dey056] [Citation(s) in RCA: 31] [Impact Index Per Article: 6.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2017] [Accepted: 03/01/2018] [Indexed: 01/22/2023] Open
Abstract
Much of the published medical research is apparently flawed, cannot be replicated and/or has limited or no utility. This article presents an overview of the current landscape of biomedical research, identifies problems associated with common study designs and considers potential solutions. Randomized clinical trials, observational studies, systematic reviews and meta-analyses are discussed in terms of their inherent limitations and potential ways of improving their conduct, analysis and reporting. The current emphasis on statistical significance needs to be replaced by sound design, transparency and willingness to share data with a clear commitment towards improving the quality and utility of clinical research.
Collapse
|
14
|
Faltinsen EG, Storebø OJ, Jakobsen JC, Boesen K, Lange T, Gluud C. Network meta-analysis: the highest level of medical evidence? BMJ Evid Based Med 2018; 23:56-59. [PMID: 29595131 DOI: 10.1136/bmjebm-2017-110887] [Citation(s) in RCA: 53] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 02/12/2018] [Indexed: 01/27/2023]
Affiliation(s)
| | - Ole Jakob Storebø
- Psychiatric Research Unit, Region Zealand Psychiatry, Slagelse, Denmark
- Department of Psychology, University of Southern Denmark, Odense, Denmark
| | - Janus C Jakobsen
- The Copenhagen Trial Unit, Centre for Clinical Intervention Research, Rigshospitalet, Copenhagen, Denmark
- Department of Cardiology, Holbaek Hospital, Holbaek, Denmark
| | - Kim Boesen
- Nordic Cochrane Centre, Rigshospitalet, Copenhagen, Denmark
| | - Theis Lange
- Section of Biostatistics, University of Copenhagen, Copenhagen, Denmark
- Centre for Statistical Science, Peking University, Beijing, China
| | - Christian Gluud
- The Copenhagen Trial Unit, Centre for Clinical Intervention Research, Rigshospitalet, Copenhagen, Denmark
| |
Collapse
|
15
|
Ioannidis JPA. Meta-analyses in environmental and occupational health. Occup Environ Med 2018; 75:443-445. [PMID: 29574405 DOI: 10.1136/oemed-2016-104128] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2016] [Revised: 02/13/2017] [Accepted: 03/15/2018] [Indexed: 12/29/2022]
Abstract
OBJECTIVES Meta-analyses are considered generally as the highest level of evidence, but concerns have been voiced about their massive, low-quality production. This paper aimed to evaluate the landscape of meta-analyses in the field of occupational and environmental health and medicine. METHODS Using relevant search terms, all meta-analyses were searched for, but those published in 2015 were assessed for their origin, whether they included randomised trials and individual-level data and whether they had authors from the industry or consultancy firms. RESULTS PubMed searches (last update February 2017) identified 1251 eligible meta-analyses in this field. There was a rapid increase over time (n=16 published in 1995 vs n=163 published in 2015). Of the 163 eligible meta-analyses published in 2015, 49 were from China, followed at a distance by the USA (n=19). Only 16 considered randomised (intervention) trials and 13 included individual-level data. Only 1 of the 150 meta-analyses had industry authors and none had consultancy firm authors. As an example of conflicting findings, 12 overlapping meta-analyses addressed mobile phones and brain cancer risk and they differed substantially in number of studies included, eligibility criteria and conclusions. CONCLUSIONS There has been a major increase in the publication of meta-analyses in occupational and environmental health over time, with the majority of these studies focusing on observational data, while a commendable fraction used individual-level data. Authorship is still limited largely to academic and non-profit authors. With massive production of meta-analyses, redundancy needs to be anticipated and efforts should be made to safeguard quality and protect from bias.
Collapse
Affiliation(s)
- John P A Ioannidis
- Department of Medicine, Stanford University School of Medicine, Stanford, California, USA.,Department of Health Research and Policy, Stanford University School of Medicine, Stanford, California, USA.,Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, California, USA.,Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, USA
| |
Collapse
|
16
|
Page MJ, Shamseer L, Tricco AC. Registration of systematic reviews in PROSPERO: 30,000 records and counting. Syst Rev 2018; 7:32. [PMID: 29463298 PMCID: PMC5819709 DOI: 10.1186/s13643-018-0699-4] [Citation(s) in RCA: 233] [Impact Index Per Article: 38.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/03/2017] [Accepted: 02/15/2018] [Indexed: 12/11/2022] Open
Abstract
BACKGROUND The International Prospective Register of Systematic Reviews (PROSPERO) was launched in February 2011 to increase transparency of systematic reviews (SRs). There have been few investigations of the content and use of the database. We aimed to investigate the number of PROSPERO registrations from inception to 2017, and website usage in the last year. We also aimed to explore the epidemiological characteristics of and completeness of primary outcome pre-specification in a sample of PROSPERO records from 2017. METHODS The PROSPERO database managers provided us with data on the annual and cumulative number of SR registrations up to October 10, 2017, and the number of visits to the PROSPERO website over the year preceding October 10, 2017. One author collected data on the focus of the SR (e.g. therapeutic, diagnostic), health area addressed, funding source and completeness of outcome pre-specification in a random sample of 150 records of SRs registered in PROSPERO between April 1, 2017 and September 30, 2017. RESULTS As of October 10, 2017, there were 26,535 SRs registered in PROSPERO; guided by current monthly submission rates, we anticipate this figure will reach over 30,000 by the end of 2017. There has been a 10-fold increase in registrations, from 63 SRs per month in 2012 to 800 per month in 2017. In the year preceding October 10, 2017, the PROSPERO website received more than 1.75 million page views. In the random sample of 150 registered SRs, the majority were focused on a therapeutic question (78/150 [52%]), while only a few focused on a diagnostic/prognostic question (11/150 [7%]). The 150 registered SRs addressed 18 different health areas. Any information about the primary outcome other than the domain (e.g. timing, effect measures) was not pre-specified in 44/150 records (29%). CONCLUSIONS Registration of SRs in PROSPERO increased rapidly between 2011 and 2017, thus benefiting users of health evidence who want to know about ongoing SRs. Further work is needed to explore how closely published SRs adhere to the planned methods, whether greater pre-specification of outcomes prevents selective inclusion and reporting of study results, and whether registered SRs address necessary questions.
Collapse
Affiliation(s)
- Matthew J Page
- School of Public Health and Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, VIC, 3004, Australia.
| | - Larissa Shamseer
- Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, K1H 8L6, Canada.,School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, K1H 8M5, Canada
| | - Andrea C Tricco
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, ON, M5B 1W8, Canada.,Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, M5T 3M7, Canada
| |
Collapse
|
17
|
Tricco AC, Zarin W, Ghassemi M, Nincic V, Lillie E, Page MJ, Shamseer L, Antony J, Rios P, Hwee J, Veroniki AA, Moher D, Hartling L, Pham B, Straus SE. Same family, different species: methodological conduct and quality varies according to purpose for five types of knowledge synthesis. J Clin Epidemiol 2017; 96:133-142. [PMID: 29103958 DOI: 10.1016/j.jclinepi.2017.10.014] [Citation(s) in RCA: 51] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2017] [Revised: 08/28/2017] [Accepted: 10/16/2017] [Indexed: 12/23/2022]
Abstract
OBJECTIVES The aim of the study was to characterize methodological conduct, reporting, and quality of five knowledge synthesis (KS) approaches. STUDY DESIGN AND SETTING Retrospective analysis of a convenience sample of five published databases of KS approaches: overview of reviews (n = 74), scoping reviews (n = 494), rapid reviews (n = 84), systematic reviews (n = 300), and network meta-analyses (NMAs; n = 456). Data in the five published databases were abstracted by two reviewers independently, any missing data for this retrospective analysis were abstracted by one experienced reviewer. Methods were appraised using the A MeaSurement Tool to Assess systematic Reviews (AMSTAR) tool. Descriptive analysis was performed. RESULTS Reporting the use of a protocol ranged from 4% for rapid reviews to 32% for systematic reviews. The use of two reviewers for citation and full-text screening ranged from 20% for scoping reviews to 60% for NMAs. Data abstraction was performed in duplicate for 11% of rapid reviews and 54% of NMAs, and for risk of bias appraisal, this ranged from 6% for scoping reviews to 41% for NMAs. NMAs had the highest median percentage of maximum obtainable AMSTAR score (64%; Q1-Q3:45-73%), while scoping reviews had the lowest (25%; Q1-Q3:13-38%). CONCLUSION NMAs consistently scored the highest on the AMSTAR tool likely because the purpose is to estimate treatment effects statistically. Scoping reviews scored the lowest (even after adjusting the score for not relevant items) likely because the purpose is to characterize the literature.
Collapse
Affiliation(s)
- Andrea C Tricco
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada; Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, 155 College Street, 6th Floor, Toronto, Ontario M5T 3M7, Canada.
| | - Wasifa Zarin
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
| | - Marco Ghassemi
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
| | - Vera Nincic
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
| | - Erin Lillie
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
| | - Matthew J Page
- School of Public Health & Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, Victoria 3004, Australia; School of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK
| | - Larissa Shamseer
- Ottawa Hospital Research Institute, The Ottawa Hospital, 501 Smyth Road, PO BOX 201B, Ottawa, Ontario K1H 8L6, Canada
| | - Jesmin Antony
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
| | - Patricia Rios
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
| | - Jeremiah Hwee
- Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, 155 College Street, 6th Floor, Toronto, Ontario M5T 3M7, Canada
| | - Areti Angeliki Veroniki
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
| | - David Moher
- Ottawa Hospital Research Institute, The Ottawa Hospital, 501 Smyth Road, PO BOX 201B, Ottawa, Ontario K1H 8L6, Canada
| | - Lisa Hartling
- School of Public Health, University of Alberta, 116 St & 85 Ave, Edmonton, Alberta T6G 2R3, Canada
| | - Ba' Pham
- Toronto Health Economics and Technology Assessment, University of Toronto, 27 Kings College Circle, Toronto, Ontario M5S 1A1, Canada
| | - Sharon E Straus
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada; Department of Geriatric Medicine, University of Toronto, 27 Kings College Circle, Toronto, Ontario M5S 1A1, Canada
| |
Collapse
|
18
|
Ioannidis J. Next-generation systematic reviews: prospective meta-analysis, individual-level data, networks and umbrella reviews. Br J Sports Med 2017; 51:1456-1458. [DOI: 10.1136/bjsports-2017-097621] [Citation(s) in RCA: 104] [Impact Index Per Article: 14.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/03/2017] [Indexed: 12/31/2022]
|