History of evidence synthesis to assess treatment effects: Personal reflections on something that is very much alive

Exploring undiscovered public knowledge in neuroscience

In this essay, I argue that the combination of research synthesis and philosophical methods can fill an important methodological gap in neuroscience. While experimental research and formal modelling have seen their methods progressively increase in rigour and sophistication over the years, the task of analysing and synthesizing the vast literature reporting new results and models has lagged behind. The problem is aggravated because neuroscience has grown and expanded into a vast mosaic of related but partially independent subfields, each with their own literatures. This fragmentation not only makes it difficult to see the full picture emerging from neuroscience research but also limits progress in individual subfields. The current neuroscience literature has the perfect conditions to create what the information scientist Don Swanson called "undiscovered public knowledge"-knowledge that exists in the mutual implications of different published pieces of information but that is nonetheless undiscovered because those pieces have not been explicitly connected. Current methods for rigorous research synthesis, such as systematic reviews and meta-analyses, mostly focus on combining similar studies and are not suited for exploring undiscovered public knowledge. To that aim, they need to be adapted and supplemented. I argue that successful exploration of the hidden implications in the neuroscience literature will require the combination of these adapted research synthesis methods with philosophical methods for rigorous (and creative) analysis and synthesis.

Interventions to promote medical student well-being: an overview of systematic reviews

OBJECTIVE

To conduct an overview of systematic reviews that explore the effectiveness of interventions to enhance medical student well-being.

DESIGN

Overview of systematic reviews.

DATA SOURCES

The Cochrane Library of Systematic Reviews, MEDLINE, APA PsychInfo, CINAHL and Scopus were searched from database inception until 31 May 2023 to identify systematic reviews of interventions to enhance medical student well-being. Ancestry searching and citation chasing were also conducted.

DATA EXTRACTION AND SYNTHESIS

The Assessing the Methodological Quality of Systematic Reviews V.2 tool was used to appraise the quality of the included reviews. A narrative synthesis was conducted, and the evidence of effectiveness for each intervention was rated.

RESULTS

13 reviews (with 94 independent studies and 17 616 students) were included. The reviews covered individual-level and curriculum-level interventions. Individual interventions included mindfulness (n=12), hypnosis (n=6), mental health programmes (n=7), yoga (n=4), cognitive and behavioural interventions (n=1), mind-sound technology (n=1), music-based interventions (n=1), omega-3 supplementation (n=1), electroacupuncture (n=1) and osteopathic manipulative treatment (n=1). The curriculum-level interventions included pass/fail grading (n=4), problem-based curriculum (n=2) and multicomponent curriculum reform (n=2). Most interventions were not supported by sufficient evidence to establish effectiveness. Eleven reviews were rated as having 'critically low' quality, and two reviews were rated as having 'low' quality.

CONCLUSIONS

Individual-level interventions (mindfulness and mental health programmes) and curriculum-level interventions (pass/fail grading) can improve medical student well-being. These conclusions should be tempered by the low quality of the evidence. Further high-quality research is required to explore additional effective interventions to enhance medical student well-being and the most efficient ways to implement and combine these for maximum benefit.

Systematic literature reviews over the years

Purpose: Nowadays, systematic literature reviews (SLRs) and meta-analyses are often placed at the top of the study hierarchy of evidence. The main objective of this paper is to evaluate the trends in SLRs of randomized controlled trials (RCTs) throughout the years. Methods: Medline database was searched, using a highly focused search strategy. Each paper was coded according to a specific ICD-10 code; the number of RCTs included in each evaluated SLR was also retrieved. All SLRs analyzing RCTs were included. Protocols, commentaries, or errata were excluded. No restrictions were applied. Results: A total of 7,465 titles and abstracts were analyzed, from which 6,892 were included for further analyses. There was a gradual increase in the number of annual published SLRs, with a significant increase in published articles during the last several years. Overall, the most frequently analyzed areas were diseases of the circulatory system (n = 750) and endocrine, nutritional, and metabolic diseases (n = 734). The majority of SLRs included between 11 and 50 RCTs each. Conclusions: The recognition of SLRs' usefulness is growing at an increasing speed, which is reflected by the growing number of published studies. The most frequently evaluated diseases are in alignment with leading causes of death and disability worldwide.

Sharing individual participant data: through a systematic reviewer lens

An increasing prevalence of data-sharing models, aimed at making individual participant data (IPD) from clinical trials widely available, should facilitate the conduct of systematic reviews and meta-analyses based on IPD. We have assessed these different data-sharing approaches, from the perspective of experienced IPD reviewers, to examine their utility for conducting systematic reviews based on IPD, and to highlight any challenges. We present an overview of the range of different models, including the traditional, single question approach, topic-based repositories, and the newer generic data platforms, and show that there are benefits and drawbacks to each. In particular, not all of the new models allow researchers to fully realise the well-documented advantages of using IPD for meta-analysis, and we offer potential solutions that can help improve both data quantity and utility. However, to achieve the “nirvana” of an ideal clinical data sharing environment, both for IPD meta-analysis and other secondary research purposes, we propose that data providers, data requestors, funders, and platforms need to adopt a more joined-up and standardised approach.

Bias against Vitamin C in Mainstream Medicine: Examples from Trials of Vitamin C for Infections

Evidence has shown unambiguously that, in certain contexts, vitamin C is effective against the common cold. However, in mainstream medicine, the views on vitamin C and infections have been determined by eminence-based medicine rather than evidence-based medicine. The rejection of the demonstrated benefits of vitamin C is largely explained by three papers published in 1975-two published in JAMA and one in the American Journal of Medicine-all of which have been standard citations in textbooks of medicine and nutrition and in nutritional recommendations. Two of the papers were authored by Thomas Chalmers, an influential expert in clinical trials, and the third was authored by Paul Meier, a famous medical statistician. In this paper, we summarize several flaws in the three papers. In addition, we describe problems with two recent randomized trial reports published in JAMA which were presented in a way that misled readers. We also discuss shortcomings in three recent JAMA editorials on vitamin C. While most of our examples are from JAMA, it is not the only journal with apparent bias against vitamin C, but it illustrates the general views in mainstream medicine. We also consider potential explanations for the widespread bias against vitamin C.

Methods used to assess outcome consistency in clinical studies: A literature-based evaluation

Evaluation studies of outcomes used in clinical research and their consistency are appearing more frequently in the literature, as a key part of the core outcome set (COS) development. Current guidance suggests such evaluation studies should use systematic review methodology as their default. We aimed to examine the methods used. We searched the Core Outcome Measures in Effectiveness Trials (COMET) database (up to May 2019) supplementing it with additional resources. We included evaluation studies of outcome consistency in clinical studies across health subjects and used a subset of A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 2 (items 1-9) to assess their methods. Of 93 included evaluation studies of outcome consistency (90 full reports, three summaries), 91% (85/93) reported performing literature searches in at least one bibliographic database, and 79% (73/93) was labelled as a "systematic review". The evaluations varied in terms of satisfying AMSTAR 2 criteria, such that 81/93 (87%) had implemented PICO in the research question, whereas only 5/93 (6%) had included the exclusions list. None of the evaluation studies explained how inconsistency of outcomes was detected, however, 80/90 (88%) concluded inconsistency in individual outcomes (66%, 55/90) or outcome domains (20%, 18/90). Methods used in evaluation studies of outcome consistency in clinical studies differed considerably. Despite frequent being labelled as a "systematic review", adoption of systematic review methodology is selective. While the impact on COS development is unknown, authors of these studies should refrain from labelling them as "systematic review" and focus on ensuring that the methods used to generate the different outcomes and outcome domains are reported transparently.

Partially systematic thoughts on the history of systematic reviews

Six years after the launch of Systematic Reviews by Biomed Central, this article is part of the celebration of the journal. It contains personal reflections on the past, present and future of systematic reviews, using examples relevant to the role of systematic reviews in cataloguing and analysing research, assessing quality and planning new studies. The focus is on the most common of the various types of systematic review in health and social care, namely those assessing the effects of interventions.

Collaboration challenges in systematic reviews: a survey of health sciences librarians

OBJECTIVE

While many librarians have been asked to participate in systematic reviews with researchers, often these researchers are not familiar with the systematic review process or the appropriate role for librarians. The purpose of this study was to identify the challenges and barriers that librarians face when collaborating on systematic reviews. To take a wider view of the whole process of collaborating on systematic reviews, the authors deliberately focused on interpersonal and methodological issues other than searching itself.

METHODS

To characterize the biggest challenges that librarians face while collaborating on systematic review projects, we used a web-based survey. The thirteen-item survey included seventeen challenges grouped into two categories: methodological and interpersonal. Participants were required to indicate the frequency and difficulty of the challenges listed. Open-ended questions allowed survey participants to describe challenges not listed in the survey and to describe strategies used to overcome challenges.

RESULTS

Of the 17 challenges listed in the survey, 8 were reported as common by over 40% of respondents. These included methodological issues around having too broad or narrow research questions, lacking eligibility criteria, having unclear research questions, and not following established methods. The remaining challenges were interpersonal, including issues around student-led projects and the size of the research team. Of the top 8 most frequent challenges, 5 were also ranked as most difficult to handle. Open-ended responses underscored many of the challenges included in the survey and revealed several additional challenges.

CONCLUSIONS

These results suggest that the most frequent and challenging issues relate to development of the research question and general communication with team members. Clear protocols for collaboration on systematic reviews, as well as a culture of mentorship, can help librarians prevent and address these challenges.

Systematic reviews and meta-analysis

Systematic reviews and meta-analyses are at the top of the 'evidence hierarchy' when assessing the effectiveness of health interventions. As such, they are important sources of synthesized information for decision-makers including consumers, clinicians, funders, payers, regulators, and researchers. The main reasons for undertaking systematic reviews and meta-analyses are to minimize bias and to maximize data by collating all the relevant, available evidence on a particular topic. In order to correctly inform decision-makers, but not mislead them, a number of key methodological conditions need to be met when undertaking these types of analysis. In this article we first review the history of systematic reviews and meta-analyses and then outline those conditions that may lead to the correct, or incorrect, use of these types of study. Also, new variations on standard systematic review methods are explored, with the pros and cons of each outlined.