Meta-analysis: statistical alchemy for the 21st century

The pseudoscience of lithium and suicide: Reanalysis of a misleading meta-analysis

By manipulating inclusion criteria, one can prove whatever point one wishes in meta-analysis. This critique examines a recent meta-analysis claiming lithium ineffectiveness for suicidality, based on three biased features: inclusion of many large studies specifically designed to exclude suicidality, producing zero suicide outcomes in all groups (n = 1856), thereby artificially decreasing statistical significance; arbitrary exclusion of all trials prior to the year 2000, thereby excluding two randomized clinical trials which demonstrated benefit for lithium; and underreporting of placebo suicide events in a recent randomized trial. It thereby created a smaller effect size (two suicides with lithium versus five with placebo = RR = 0.42), though still beneficial for lithium, and a larger denominator of no events (total n for included studies = 2578), leading to the claim of statistical non-significance (95% confidence intervals (CIs) 0.1-4.5). The same literature can be analyzed including the two excluded older studies, and including the two placebo deaths in the recent trial, producing a larger effect size (two suicides with lithium versus nine with placebo, RR = 0.25). Furthermore, uninformative studies with no events could be excluded (total n for included studies = 1203), as is standard practice in meta-analysis, producing statistically significant results (95% CIs 0.05, 0.83). This more complete, more accurate, and less biased meta-analysis is provided in this article.In short, including all studies with non-zero suicide outcomes, there is clear benefit for lithium. The recent meta-analysis is a classic example of pseudoscience, using scientific technique superficially to confirm, rather than refute, one's own opinions.

Structured critical analysis correction of a systematic review

OBJECTIVE

: Systematic reviews (SRs) are an increasingly important format in the scientific literature. Commentaries on improvements to the SR format have focused on methodological quality, but a greater concern is a frequent lack of critical analysis. A structured critical analysis (SCA) was described as a solution to this deficiency.

METHODS

: Recommendations and conclusions of a recent SR were analyzed with a SCA to address common problems previously reported with the SR format.

RESULTS

: Errors in the component studies and their interpretation by the SR that led to erroneous recommendations were presented. The 5-part SCA provided comprehensive analysis that corrected the SR, which had accepted the component study conclusions verbatim.

CONCLUSION

: The SCA is a logical approach to provide critical thinking in SRs to ensure appropriate conclusions. This is especially important, as many SRs report contradictory evidence. Also, the reader can use the SCA format to better understand existing literature.

Biological Reductionism as an Obstacle to the Advancement of the Biopsychosocial Concept of Mental Disorders

The substantial progress in neurobiological technologies has narrowed the horizons of many psychiatrists, ultimately leading them to focus exclusively on biomedical research, primarily aimed at studying the biological basis of mental illnesses. This has led to an unjustified dominance of the biomedical paradigm in understanding the nature of mental disorders, while virtually ignoring the study of other components of the disease related to the psychosocial maladjustment of patients. This trend, largely associated with advancements in neuroscience employing neuroimaging techniques to study the brain's activity as a biophysical object, has contributed to the development of such innovative field as evidence-based medicine. The methods of evidence-based medicine are seen as adequate in terms of determining the effectiveness of therapy for predominantly biologically determined components of mental illness (including the selection of medications) and only partially for psychological interventions. However, it seems that the predominant use of evidence-based medicine principles is insufficient for a holistic diagnostic approach, which includes a multilevel (diversified) representation of the criteria of effectiveness for pharmacological and psychological interventions. In this regard, it is promising to establish a scientifically and clinically productive combination of, on the one hand, the evidence-based concept of effectiveness assessments based on high-quality randomized scientific studies, and on the other, expert opinions of highly qualified scientific specialists, as well as practicing physicians with their personal professional experience in individualized therapy. This makes it reasonable to develop a personality-oriented personalized psychiatry, based on a biopsychosocial understanding of the nature of mental disorders, their holistic assessment, and the development of comprehensive therapeutic measures.

A statistical method for removing unbalanced trials with multiple covariates in meta-analysis

In meta-analysis literature, there are several checklists describing the procedures necessary to evaluate studies from a qualitative point of view, whereas preliminary quantitative and statistical investigations on the "combinability" of trials have been neglected. Covariate balance is an important prerequisite to conduct meta-analysis. We propose a method to identify unbalanced trials with respect to a set of covariates, in presence of covariate imbalance, namely when the randomized controlled trials generate a meta-sample that cannot satisfy the requisite of randomization/combinability in meta-analysis. The method is able to identify the unbalanced trials, through four stages aimed at achieving combinability. The studies responsible for the imbalance are identified, and then they can be eliminated. The proposed procedure is simple and relies on the combined Anderson-Darling test applied to the Empirical Cumulative Distribution Functions of both experimental and control meta-arms. To illustrate the method in practice, two datasets from well-known meta-analyses in the literature are used.

Replicating a Meta-Analysis: The Search for the Optimal Word Choice Test Cutoff Continues

This study was designed to expand on a recent meta-analysis that identified ≤42 as the optimal cutoff on the Word Choice Test (WCT). We examined the base rate of failure and the classification accuracy of various WCT cutoffs in four independent clinical samples (N = 252) against various psychometrically defined criterion groups. WCT ≤ 47 achieved acceptable combinations of specificity (.86-.89) at .49 to .54 sensitivity. Lowering the cutoff to ≤45 improved specificity (.91-.98) at a reasonable cost to sensitivity (.39-.50). Making the cutoff even more conservative (≤42) disproportionately sacrificed sensitivity (.30-.38) for specificity (.98-1.00), while still classifying 26.7% of patients with genuine and severe deficits as non-credible. Critical item (.23-.45 sensitivity at .89-1.00 specificity) and time-to-completion cutoffs (.48-.71 sensitivity at .87-.96 specificity) were effective alternative/complementary detection methods. Although WCT ≤ 45 produced the best overall classification accuracy, scores in the 43 to 47 range provide comparable objective psychometric evidence of non-credible responding. Results question the need for designating a single cutoff as "optimal," given the heterogeneity of signal detection environments in which individual assessors operate. As meta-analyses often fail to replicate, ongoing research is needed on the classification accuracy of various WCT cutoffs.

10 years of Bayesian theories of autism: A comprehensive review

Ten years ago, Pellicano and Burr published one of the most influential articles in the study of autism spectrum disorders, linking them to aberrant Bayesian inference processes in the brain. In particular, they proposed that autistic individuals are less influenced by their brains' prior beliefs about the environment. In this systematic review, we investigate if this theory is supported by the experimental evidence. To that end, we collect all studies which included comparisons across diagnostic groups or autistic traits and categorise them based on the investigated priors. Our results are highly mixed, with a slight majority of studies finding no difference in the integration of Bayesian priors. We find that priors developed during the experiments exhibited reduced influences more frequently than priors acquired previously, with various studies providing evidence for learning differences between participant groups. Finally, we focus on the methodological and computational aspects of the included studies, showing low statistical power and often inconsistent approaches. Based on our findings, we propose guidelines for future research.

Computer clinical decision support that automates personalized clinical care: a challenging but needed healthcare delivery strategy

How to deliver best care in various clinical settings remains a vexing problem. All pertinent healthcare-related questions have not, cannot, and will not be addressable with costly time- and resource-consuming controlled clinical trials. At present, evidence-based guidelines can address only a small fraction of the types of care that clinicians deliver. Furthermore, underserved areas rarely can access state-of-the-art evidence-based guidelines in real-time, and often lack the wherewithal to implement advanced guidelines. Care providers in such settings frequently do not have sufficient training to undertake advanced guideline implementation. Nevertheless, in advanced modern healthcare delivery environments, use of eActions (validated clinical decision support systems) could help overcome the cognitive limitations of overburdened clinicians. Widespread use of eActions will require surmounting current healthcare technical and cultural barriers and installing clinical evidence/data curation systems. The authors expect that increased numbers of evidence-based guidelines will result from future comparative effectiveness clinical research carried out during routine healthcare delivery within learning healthcare systems.

Artificial Intelligence and the Risk for Intuition Decline in Clinical Medicine

Artificial intelligence (AI) is revolutionizing big data analytics. In this issue of The American Journal of Gastroenterology, Ahn et al. introduce the AI-cirrhosis-electrocardiogram score that can grade the electrophysiologic cardiac changes present in patients with cirrhosis. Apart from showing excellent accuracy to identify cirrhosis, the AI-cirrhosis-electrocardiogram algorithm identified a biological gradient and signal reversibility after transplantation. Clinical applicability needs to be determined. Some concerns inherent to the use of AI are discussed, including the need to verify that the quality of data used for machine training is optimal. The black box nature of AI-identified associations is discussed, along with the lack of pathophysiologic coherence allowing intuitive medical reasoning.

Is meta-analysis of RCTs assessing the efficacy of interventions a reliable source of evidence for therapeutic decisions?

Literature-based meta-analysis is a standard technique applied to pool results of individual studies used in medicine and social sciences. It has been criticized for being too malleable to constrain results, averaging incomparable values, lacking a measure of evidence's strength, and problems with a systematic bias of individual studies. We argue against using literature-based meta-analysis of RCTs to assess treatment efficacy and show that therapeutic decisions based on meta-analytic average are not optimal given the full scope of existing evidence. The argument proceeds with discussing examples and analyzing the properties of some standard meta-analytic techniques. First, we demonstrate that meta-analysis can lead to reporting statistically significant results despite the treatment's limited efficacy. Second, we show that meta-analytic confidence intervals are too narrow compared to the variability of treatment outcomes reported by individual studies. Third, we argue that literature-based meta-analysis is not a reliable measurement instrument. Finally, we show that meta-analysis averages out the differences among studies and leads to a loss of information. Despite these problems, literature-based meta-analysis is useful for the assessment of harms. We support two alternative approaches to evidence amalgamation: meta-analysis of individual patient data (IPD) and qualitative review employing mechanistic evidence.

Fixed- and Random-Effects Models

Deciding whether to use a fixed-effect model or a random-effects model is a primary decision an analyst must make when combining the results from multiple studies through meta-analysis. Both modeling approaches estimate a single effect size of interest. The fixed-effect meta-analysis assumes that all studies share a single common effect and, as a result, all of the variance in observed effect sizes is attributable to sampling error. The random-effects meta-analysis estimates the mean of a distribution of effects, thus assuming that study effect sizes vary from one study to the next. Under this model, variance in observed effect sizes is attributable to both sampling error (within-study variance) and statistical heterogeneity (between-study variance).The most popular meta-analyses involve using a weighted average to combine the study-level effect sizes. Both fixed- and random-effects models use an inverse-variance weight (variance of the observed effect size). However, given the shared between-study variance used in the random-effects model, it leads to a more balanced distribution of weights than under the fixed-effect model (i.e., small studies are given more relative weight and large studies less). The standard error for these estimators also relates to the inverse-variance weights. As such, the standard errors and confidence intervals for the random-effects model are larger and wider than in the fixed-effect analysis. Indeed, in the presence of statistical heterogeneity, fixed-effect models can lead to overly narrow intervals.In addition to commonly used, generalizable models, there are additional fixed-effect models and random-effect models that can be considered. Additional fixed-effect models that are specific to dichotomous data are more robust to issues that arise from sparse data. Furthermore, random-effects models can be expanded upon using generalized linear mixed models so that different covariance structures are used to distribute statistical heterogeneity across multiple parameters. Finally, both fixed- and random-effects modeling can be conducted using a Bayesian framework.

A Rational Use of Clozapine Based on Adverse Drug Reactions, Pharmacokinetics, and Clinical Pharmacopsychology

Using Richardson and Davidson's model and the sciences of pharmacokinetics and clinical pharmacopsychology, this article reviewed the: (1) poor life expectancy associated with treatment-resistant schizophrenia (TRS), which may be improved in patients who adhere to clozapine; (2) findings that clozapine is the best treatment for TRS (according to efficacy, effectiveness and well-being); and (3) potential for clozapine to cause vulnerabilities, including potentially lethal adverse drug reactions such as agranulocytosis, pneumonia, and myocarditis. Rational use requires: (1) modification of the clozapine package insert worldwide to include lower doses for Asians and to avoid the lethality associated with pneumonia, (2) the use of clozapine levels for personalizing dosing, and (3) the use of slow and personalized titration. This may make clozapine as safe as possible and contribute to increased life expectancy and well-being. In the absence of data on COVID-19 in clozapine patients, clozapine possibly impairs immunological mechanisms and may increase pneumonia risk in infected patients. Psychiatrists should call their clozapine patients and families and explain to them that if the patient develops fever or flu-like symptoms, the psychiatrist should be called and should consider halving the clozapine dose. If the patient is hospitalized with pneumonia, the treating physician needs to assess for symptoms of clozapine intoxication since halving the dose may not be enough for all patients; consider decreasing it to one-third or even stopping it. Once the signs of inflammation and fever have disappeared, the clozapine dose can be slowly increased to the prior dosage level.

The Usefulness of Meta-Analyses to Hip and Knee Surgeons

BACKGROUND

Comprehensive systematic reviews of results from homogenous or heterogeneous clinical trials, meta-analyses are used to summarize and to interpret studies. Proponents believe that their use can increase study power and improve precision results. Critics emphasize that heterogeneity between studies and bias of individual studies compromise the value of results. The methodology of meta-analyses has improved over time, utilizing statistical analysis to reduce bias and examining heterogeneity. With an increasing trend of meta-analyses in orthopaedic literature, this study aimed to investigate quality and clinical utility of meta-analyses for total knee arthroplasty and total hip arthroplasty.

METHODS

A systematic review of total knee arthroplasty and total hip arthroplasty meta-analyses in 3 major orthopaedic journals from January 2000 to August 2017 was performed. Three authors independently reviewed eligible meta-analyses. A quality assessment was conducted using the Oxman-Guyatt Index. Reporting quality was assessed using PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). Two high-volume, fellowship-trained, attending surgeons specializing in total hip arthroplasty and total knee arthroplasty independently, in a blinded fashion, reviewed 24 of the highest-scored meta-analyses.

RESULTS

There were 114 studies meeting eligibility criteria, 25 published from 2000 to 2009 and 89 published from 2010 to 2017, a 3.6-fold increase. The mean Oxman-Guyatt Index score was 3.89 points, with 12 high-quality studies, 87 moderate-quality studies, and 15 low-quality studies. The mean PRISMA score for all meta-analyses was 22.2 points, with 79% classified as low to moderate. Only 23 studies listed the Level of Evidence, and 8 were Level-I studies and 9 were Level-II studies. Studies with >15 randomized controlled trials were associated with higher PRISMA and Oxman-Guyatt Index scores. In 12 articles, we were unable to decipher the types of studies included. Only 39.4% of studies showed the risk of bias. Of the 24 studies identified as high quality per the PRISMA statement, 71% were determined to be either clinically unimportant or inconclusive.

CONCLUSIONS

The number of total hip arthroplasty and total knee arthroplasty meta-analyses has markedly increased over the past decades. The majority of published meta-analyses from 3 major orthopaedic journals were not performed in accordance with established PRISMA guidelines.

CLINICAL RELEVANCE

Many published meta-analyses are low to moderate quality, and clinicians should cautiously draw conclusions from poorly executed meta-analyses.

Interventions and cognitive functioning in adults with traumatic spinal cord injuries: a systematic review and meta-analysis

AIM

This research synthesized scientific evidence on the impact of interventions for adults with traumatic spinal cord injury on cognition, to understand if current intervention approaches are appropriate in light of the risk of post-injury cognitive impairments.

METHOD

Medline, Central, Embase, Scopus, PsycINFO and PubMed were searched for intervention in persons with SCI assessing cognition pre- and post-intervention. Study quality was completed using the National Institutes of Health quality assessment tools. Results were grouped by type of intervention. The meta-analysis involved calculation of pooled effect sizes for interventions utilizing the same cognitive measure.

RESULTS

Eleven studies of moderate quality discussed drug therapy, transcutaneous tibial nerve stimulation, diet modification and dietary supplements, and inpatient rehabilitation. Some aspects of cognition were negatively affected by drugs while diet modification and supplement use, and transcutaneous tibial nerve stimulation showed no evidence of a difference in cognitive scores when compared with no intervention. Inpatient rehabilitation revealed a small but beneficial effect, when results of seven studies were pooled.

CONCLUSION

Evidence on the effects of interventions on cognitive functioning in patients with traumatic spinal cord injury is sparse and inconclusive, so work in this area is timely. It is valuable to know not only which interventions are effective for improving cognition, but also how other commonly used interventions, intended to treat other injury sequela, can affect cognition. PROSPERO: CRD42018087238.Implications for rehabilitationHistorically, rehabilitation of patients with traumatic spinal cord injury has targeted physical impairments, with little attention to cognition; this research aimed to understand if current interventions are appropriate in light of the risk of patients' cognitive impairments. Evidence on the effects of drug therapy, diet and dietary supplements interventions on cognitive functioning in traumatic spinal cord injury is sparse and inconclusive.Combining multiple inpatient rehabilitation interventions shows a positive but heterogeneous effect on the cognitive functioning; interventions applied earlier show greater gains.A major challenge for clinicians is to select an outcome measure sensitive to change over time, and to relate the results to patients' change in cognitive abilities with intervention applicationResearch to understand the functional effect of spinal cord injury on the widely distributed networks of the central and autonomic nervous systems subserving cognition, is timely.

Long-term Concentrations of Nitrogen Dioxide and Mortality: A Meta-analysis of Cohort Studies

Supplemental Digital Content is available in the text.

Background:

Concentrations of outdoor nitrogen dioxide (NO₂) have been associated with increased mortality. Hazard ratios (HRs) from cohort studies are used to assess population health impact and burden. We undertook meta-analyses to derive concentration–response functions suitable for such evaluations and assessed their sensitivity to study selection based upon cohort characteristics.

Methods:

We searched online databases and existing reviews for cohort studies published to October 2016 that reported HRs for NO₂ and mortality. We calculated meta-analytic summary estimates using fixed/random-effects models.

Results:

We identified 48 articles analyzing 28 cohorts. Meta-analysis of HRs found positive associations between NO₂ and all cause (1.02 [95% confidence interval (CI): 1.01, 1.03]; prediction interval [PI]: [0.99, 1.06] per 10 µg/m³ increment in NO₂), cardiovascular (1.03 [95% CI: 1.02, 1.05]; PI: [0.98, 1.08]), respiratory (1.03 [95% CI: 1.01, 1.05]; PI: [0.97, 1.10]), and lung cancer mortality (1.05 [95% CI: 1.02, 1.08]; PI: [0.94, 1.17]) with evidence of substantial heterogeneity between studies. In subgroup analysis, summary HRs varied by age at cohort entry, spatial resolution of pollution estimates, and adjustment for smoking and body mass index at the individual level; for some subgroups, the HR was close to unity, with lower confidence limits below 1.

Conclusions:

Given the many uncertainties inherent in the assessment of this evidence base and the sensitivity of health impact calculations to small changes in the magnitude of the HRs, calculation of the impact on health of policies to reduce long-term exposure to NO₂ should use prediction intervals and report ranges of impact rather than focusing upon point estimates.

Clinical and Health System Determinants of Venous Thromboembolism Event Rates After Hip Arthroplasty: An International Comparison

BACKGROUND

Routinely collected hospital data provide increasing opportunities to assess the performance of health care systems. Several factors may, however, influence performance measures and their interpretation between countries.

OBJECTIVE

We compared the occurrence of in-hospital venous thromboembolism (VTE) in patients undergoing hip replacement across 5 countries and explored factors that could explain differences across these countries.

METHODS

We performed cross-sectional studies independently in 5 countries: Canada; France; New Zealand; the state of California; and Switzerland. We first calculated the proportion of hospital inpatients with at least one deep vein thrombosis (DVT) or pulmonary embolism by using numerator codes from the corresponding Patient Safety Indicator. We then compared estimates from each country against a reference value (benchmark) that displayed the baseline risk of VTE in such patients. Finally, we explored length of stay, number of secondary diagnoses coded, and systematic use of ultrasound to detect DVT as potential factors that could explain between-country differences.

RESULTS

The rates of VTE were 0.16% in Canada, 1.41% in France, 0.84% in New Zealand, 0.66% in California, and 0.37% in Switzerland, while the benchmark was 0.58% (95% confidence interval, 0.35-0.81). Factors that could partially explain differences in VTE rates between countries were hospital length of stay, number of secondary diagnoses coded, and proportion of patients who received lower limb ultrasound to screen for DVT systematically before hospital discharge. An exploration of the French data showed that the systematic use of ultrasound may be associated with over detection of DVT but not pulmonary embolism.

CONCLUSIONS

In-hospital VTE rates after arthroplasty vary widely across countries, and a combination of clinical, data-related, and health system factors explain some of the variations in VTE rates across countries.

Summarising the Evidence for Drug Safety: A Methodological Discussion of Different Meta-Analysis Approaches

Evidence on drug safety obtained from randomised clinical trials is very limited due to, among other reasons, their relatively small sample size. Hence, combining the results of available studies can prove particularly useful. This paper reviews the different data sources for summarising drug safety outcomes, according to study design, publication of data, and origin of the information. It then discusses the various types of overviews that can be used in the study of treatment harms, focusing on meta-analyses of aggregate data and meta-analyses of individual patient data, with their advantages and drawbacks, such as publication bias and heterogeneity. Although the different approaches available for combining the results are of great utility in assessing treatment harms, none of them is free from limitations. Therefore, it might be appropriate to perform an analysis of sensitivity to assess whether the results are sensitive to the technique that has been used.

Acceptance of a systematic review as a thesis: survey of biomedical doctoral programs in Europe

BACKGROUND

Systematic reviews (SRs) have been proposed as a type of research methodology that should be acceptable for a graduate research thesis. The aim of this study was to analyse whether PhD theses in European biomedical graduate programs can be partly or entirely based on SRs.

METHODS

In 2016, we surveyed individuals in charge of European PhD programs from 105 institutions. The survey asked about acceptance of SRs as the partial or entire basis for a PhD thesis, their attitude towards such a model for PhD theses, and their knowledge about SR methodology.

RESULTS

We received responses from 86 individuals running PhD programs in 68 institutions (institutional response rate of 65%). In 47% of the programs, SRs were an acceptable study design for a PhD thesis. However, only 20% of participants expressed a personal opinion that SRs meet the criteria for a PhD thesis. The most common reasons for not accepting SRs as the basis for PhD theses were that SRs are 'not a result of a PhD candidate's independent work, but more of a team effort' and that SRs 'do not produce enough new knowledge for a dissertation'. The majority of participants were not familiar with basic concepts related to SRs; questions about meta-analyses and the type of plots frequently used in SRs were correctly answered by only one third of the participants.

CONCLUSIONS

Raising awareness about the importance of SRs and their methodology could contribute to higher acceptance of SRs as a type of research that forms the basis of a PhD thesis.

Con: Meta-analysis: some key limitations and potential solutions

Meta-analysis, a statistical combination of results of several trials to produce a summary effect, has been subject to criticism in the past, mainly for the reasons of poor quality of included studies, heterogeneity between studies meta-analyzed and failing to address publication bias. These limitations can cause the results to be misleading, which is important if policy and practice decisions are based on systematic reviews and meta-analyses. We elaborate on these limitations and illustrate them with examples from the nephrology literature. Finally, we present some potential solutions, notably, education in meta-analysis for evidence producers and consumers as well as the use of individual patient data for meta-analyses.

Reviving Cochrane's contribution to evidence-based medicine: bridging the gap between evidence of efficacy and evidence of effectiveness and cost-effectiveness

Throughout the quarter century since the advent of evidence-based medicine (EBM), medical research has prioritized 'efficacy' (i.e. internal validity) using randomized controlled trials. EBM has consistently neglected 'effectiveness' and 'cost-effectiveness', identified in the pioneering work of Archie Cochrane as essential for establishing the external (i.e. clinical) validity of health care interventions. Neither Cochrane nor other early pioneers appear to have foreseen the extent to which EBM would be appropriated by the pharmaceutical and medical devices industries, which are responsible for extensive biases in clinical research due to selective reporting, exaggeration of benefits, minimization of risks, and misrepresentation of data. The promise of EBM to effect transformational change in health care will remain unfulfilled until (i) studies of effectiveness and cost-effectiveness are pursued with some of the same fervour that previously succeeded in elevating the status of the randomized controlled trial, and (ii) ways are found to defeat threats to scientific integrity posed by commercial conflicts of interest.

Evidence-based medicine was bound to fail: a report to Alvan Feinstein

John Ioannidis has provided a lucid account, in the form of a report to David Sackett, of how evidence-based medicine (EBM) was hijacked to serve vested interests: major randomized controlled trials are largely done by and for the benefit of the industry; meta-analyses and guidelines are flooded with conflicts of interest; national and federal research funds are unable to address basic clinical questions. Nonetheless, EBM would remain a worthwhile goal. In this paper, in the form of a report to Alvan Feinstein, it is argued that current developments were largely predictable. EBM certainly gave an important contribution to questioning unsubstantiated therapeutic claims. Time has come, however, to become aware of its considerable limitations, including overall reductionism and insufficient consideration of problems related to financial conflicts of interest. EBM does not represent the scientific approach to medicine: it is only a restrictive interpretation of the scientific approach to clinical practice. EBM drives the prescribing clinician to an overestimated consideration of potential benefits, paying little attention to the likelihood of responsiveness and to potential vulnerabilities in relations to the adverse effects of treatment. It is time to substitute the fashionable popularity of a strategy developed outside of clinical medicine with models and research based on the insights of clinical judgment and patient-doctor interaction, as Feinstein had outlined.

Are diabetes guidelines truly evidence based?

The global epidemic of obesity and diabetes underscores the urgency to develop strategies to prevent cardiovascular (CV) disease in this vulnerable population. Clinical guidelines are intended to help the clinician manage these patients, but guidelines are often discordant among professional organizations and not always evidence based. Clinicians must rely upon the best available evidence, and therefore we critically reviewed the evidence behind the American Diabetes Association (ADA) 2016 guidelines on the prevention of CV disease in diabetes. We believe the most robust evidence comes from randomized controlled trials specifically designed for diabetes with hard clinical endpoints such as mortality and CV events. Our analysis supports the ADA recommendations regarding a Mediterranean diet, glycemic control, and BP control, but we believe the evidence to support aspirin and statin therapy in diabetes is inconclusive. This discordance may be multi-factorial including the exclusion of some relevant studies and an over-reliance upon subgroup and meta-analysis. Given the lack of mortality benefit and inconsistent clinical benefits of aspirin and statins, it is essential that clinicians individualize treatment decisions while carefully weighing the risks and harms of any intervention.

Introduction to systematic reviews and meta-analysis

Systematic reviews provide a method for collating and synthesizing research, and are used to inform healthcare decision making by clinicians, consumers and policy makers. A core component of many systematic reviews is a meta-analysis, which is a statistical synthesis of results across studies. In this review article, we introduce meta-analysis, focusing on the different meta-analysis models, their interpretation, how a model should be selected and discuss potential threats to the validity of meta-analyses. We illustrate the application of meta-analysis using data from a review examining the effects of early use of inhaled corticosteroids in the emergency department treatment of acute asthma.

Evidence-based toxicology: a comprehensive framework for causation

This paper identifies deficiencies in some current practices of causation and risk evaluation by toxicologists and formulates an evidence-based solution. The practice of toxicology focuses on adverse health events caused by physical or chemical agents. Some relations between agents and events are identified risks, meaning unwanted events known to occur at some frequency. However, other relations that are only possibilities – not known to occur (and may never be realized) – also are sometimes called risks and are even expressed quantitatively. The seemingly slight differences in connotation among various uses of the word ‘risk’ conceal deeply philosophic differences in the epistemology of harm. We label as ‘nomological possibilities’ (not as risks) all predictions of harm that are known not to be physically or logically impossible. Some of these nomological possibilities are known to be causal. We term them ‘epistemic’. Epistemic possibilities are risks. The remaining nomological possibilities are called ‘uncertainties’. Distinguishing risks (epistemic relationships) from among all nomological possibilities requires knowledge of causation. Causality becomes knowable when scientific experiments demonstrate, in a strong, consistent (repeatable), specific, dose-dependent, coherent, temporal and predictive manner that a change in a stimulus determines an asymmetric, directional change in the effect. Many believe that a similar set of characteristics, popularly called the ‘Hill Criteria’, make it possible, if knowledge is robust, to infer causation from only observational (nonexperimental) studies, where allocation of test subjects or items is not under the control of the investigator.

Until the 1980s, medical decisions about diagnosis, prevention, treatment or harm were often made authoritatively. Rather than employing a rigorous evaluation of causal relationships and applying these criteria to the published knowledge, the field of medicine was dominated by authority-based opinions, expressed by experts (or consensus groups of experts) relying on their education, training, experience, wisdom, prestige, intuition, skill and improvisation. In response, evidence-based medicine (EBM) was developed, to make a conscientious, explicit and judicious use of current best evidence in deciding about the care of individual patients. Now globally embraced, EBM employs a structured, ‘transparent’ protocol for carrying out a deliberate, objective, unbiased and systematic review of the evidence about a formally framed question. Not only in medicine, but now in dentistry, engineering and other fields that have adapted the methods of EBM, it is the quality of the evidence and the rigor of the analysis through evidence-based logic (EBL), rather than the professional standing of the reviewer, that leads to evidence-based conclusions about what is known. Recent studies have disclosed that toxicologists (individually or in expert groups), not unlike their medical counterparts prior to EBM, show distressing variations in their biases with regard to data selection, data interpretation and data evaluation when performing reviews for causation analyses. Moreover, toxicologists often fail to acknowledge explicitly (particularly in regulatory and policymaking arenas) when shortcomings in the evidence necessitate reliance upon authority-based opinions, rather than evidence-based conclusions (Guzelian PS, Guzelian CP. Authority-based explanation. Science 2004; 303: 1468-69). Accordingly, for answering questions about general and specific causation, we have constructed a framework for evidence-based toxicology (EBT), derived from the accepted principles of EBM and expressed succinctly as three stages, comprising 12 total steps. These are: 1) collecting and evaluating the relevant data (Source, Exposure, Dose, Diagnosis); 2) collecting and evaluating the relevant knowledge (Frame the question, Assemble the relevant (delimited) literature, Assess and critique the literature); and 3) Joining data with knowledge to arrive at a conclusion (General causation – answer to the framed question, Dose-response, Timing, Alternative cause, Coherence). The second of these stages (which amounts to an analysis of general causation), is addressed by an EBM-styled approach (adapted for the infrequent availability of human experimental studies in environmental toxicology). This involves assembling literature (through documented algorithms for database queries), excluding irrelevancies by use of delimiters as filters, and ranking and rating the remaining articles for strength of study design and for quality of execution gauged by application of either a ready-made quality assessment instrument or a custom designed checklist or scale. The results of this systematic review (including a structured review of relevant animal and in vitro studies) are then themselves systematically used to determine which causation criteria are fulfilled.

Toxicology is maturing from a derivative science largely devoted to routinized performance and interpretation of safety tests, to a discipline deeply enmeshed in the remarkable advances in biochemistry and molecular biology to better understanding the nature and mechanism of adverse effects caused by chemicals. It is time for toxicologists, like scientists in other fields, to formalize a method for differentiating settled toxicological knowledge of risk from mere nomological possibility, and for communicating their conclusions to other scientists and the public. It is time for EBT.

Psychology, replication & beyond

Modern psychology is apparently in crisis and the prevailing view is that this partly reflects an inability to replicate past findings. If a crisis does exists, then it is some kind of ‘chronic’ crisis, as psychologists have been censuring themselves over replicability for decades. While the debate in psychology is not new, the lack of progress across the decades is disappointing. Recently though, we have seen a veritable surfeit of debate alongside multiple orchestrated and well-publicised replication initiatives. The spotlight is being shone on certain areas and although not everyone agrees on how we should interpret the outcomes, the debate is happening and impassioned. The issue of reproducibility occupies a central place in our whig history of psychology.

Pediatric Quality Improvement: Practical and Scholarly Considerations

This article describes important aspects of health-care quality, quality improvement (QI), patient safety (PS), and approaches to research on QI/PS efforts. Common terminology to facilitate an understanding of QI and PS research is reviewed. Models for understanding system and process performance are discussed. Introductory considerations to QI data and QI research analytical considerations are provided.

Adaptive Measurement and Assessment

Adaptive approaches to measurement and assessment have been useful in organizational science for more than 100 years. Advances in psychometric theory and inexpensive computing power have propelled the field into a renaissance for every type of construct and level of analysis imaginable. Exciting innovations include the use of mobile computer-adaptive testing (CAT); expert systems (e.g., automatic item generation); and unobtrusive adaptive measurement in social media, intelligent tutoring systems, and virtual worlds. Adaptive approaches are setting the stage to better embed measurement and intervention into naturalistic organizational settings and portend substantial improvements in cross-level and longitudinal tests of organizational psychology and organizational behavior (OP/OB) hypotheses.

Understanding and evaluating meta-analysis

BACKGROUND

Meta-analysis refers to statistical methodology used to combine data from many studies to obtain an overall assessment of disease risk or treatment outcomes. In this article, the authors review basic methods, interpretation, and limitations of meta-analysis.

METHODS

Investigators use meta-analysis approaches to combine data from available studies to obtain an answer to a specific question. An investigator uses a fixed model if there is homogeneity among the combined studies and a random-effects model if there is heterogeneity. The random-effects model results in wider confidence limits and more conservative estimates of overall results. A meta-analysis can be biased because studies with negative results (no differences in treatment outcomes) are less likely to be published (publication bias).

RESULTS

A meta-analysis should include a well-specified and reproducible set of procedures, including description of data abstraction procedures, attempts to include unpublished studies, and appropriate statistical analysis that includes thorough consideration of heterogeneity and potential bias.

CONCLUSIONS

Meta-analysis cannot correct shortcomings of existing studies or data. However, if potential pitfalls are recognized, meta-analysis can be a useful tool for summarizing existing studies, providing a means to address conflicting reports. Meta-analysis can lead to increased precision, providing greater power to detect existing relationships or treatment effects. Furthermore, meta-analysis may make it possible to address questions that cannot be answered by means of individual studies.

PRACTICAL IMPLICATIONS

Meta-analysis provides an objective, quantitative synthesis of available studies but needs to be understood and assessed critically by those who use it to assess risk or make treatment decisions.

A 'reluctant' critical review: 'Manual for evidence-based clinical practice (2015)'

BACKGROUND

The Users' Guides to the Medical Literature Manual has been a major influence on the teaching and practice of health care globally.

METHODS

The 3rd edition of the multi-authored Manual was reviewed using the principles outlined in Evidence-based Medicine (EBM) texts. One 'clinical scenario' was selected for critical appraisal, as were several chapters; objectivity was enhanced by citing references to support opinions. RESULTS (SUMMARY OF THE APPRAISAL): (1) Strengths: Clinical pearls, too numerous to list.

EXAMPLES

(i) evidence is never enough to drive clinical decision making; (ii) do not rush to adopt new interventions; and (iii) question efficacy data based only on surrogate markers. (2) Weaknesses: The Manual shares shortcomings of textbooks discussed by Straus et al.: (i) references may not be current, important ones may be excluded and citations may be selective; (ii) often, opinion-based; and (iii) delays between revisions. (3) Notable omissions: Little or no discussion of: (i) important segments of the population: those <18 years of age, >65 years of age and those with multimorbidity; (ii) surgical disciplines; (iii) Greenhalgh et al.'s essay on EBM; (iv) alternate views on the hierarchy of evidence; and (vi) critical thinking. (4) Additional issues: (i) Omission of important references on dabigatran (clinical scenario: chapter 13.1); (ii) authors' advice (Chapter 13.3) to 'bypass the discussion section of published research'; and (iii) the advocacy of pre-appraised sources of evidence and network meta-analysis without warnings about limitations, are critiqued.

CONCLUSION

The Manual has several clinical pearls but readers should also be aware of shortcomings.

Field-wide meta-analyses of observational associations can map selective availability of risk factors and the impact of model specifications

OBJECTIVES

Instead of evaluating one risk factor at a time, we illustrate the utility of "field-wide meta-analyses" in considering all available data on all putative risk factors of a disease simultaneously.

STUDY DESIGN AND SETTING

We identified studies on putative risk factors of pterygium (surfer's eye) in PubMed, EMBASE, and Web of Science. We mapped which factors were considered, reported, and adjusted for in each study. For each putative risk factor, four meta-analyses were done using univariate only, multivariate only, preferentially univariate, or preferentially multivariate estimates.

RESULTS

A total of 2052 records were screened to identify 60 eligible studies reporting on 65 putative risk factors. Only 4 of 60 studies reported both multivariate and univariate regression analyses. None of the 32 studies using multivariate analysis adjusted for the same set of risk factors. Effect sizes from different types of regression analyses led to significantly different summary effect sizes (P-value < 0.001). Observed heterogeneity was very high for both multivariate (median I(2), 76.1%) and univariate (median I(2), 85.8%) estimates. No single study investigated all 11 risk factors that were statistically significant in at least one of our meta-analyses.

CONCLUSION

Field-wide meta-analyses can map availability of risk factors and trends in modeling, adjustments and reporting, as well as the impact of differences in model specification.

Randomized controlled trials and neuro-oncology: should alternative designs be considered?

Deficiencies in design and reporting of randomized controlled trials (RCTs) hinders interpretability and critical appraisal. The reporting quality of recent RCTs in neuro-oncology was analyzed to assess adequacy of design and reporting. The MEDLINE and EMBASE databases were searched to identify non-surgical RCTs (years 2005-2014, inclusive). The CONSORT and Jadad scales were used to assess the quality of design/reporting. Studies published in 2005-2010 were compared as a cohort against studies published in 2011-2014, in terms of general characteristics and reporting quality. A PRECIS-based scale was used to designate studies on the pragmatic-explanatory continuum. Spearman's test was used to assess correlations. Regression analysis was used to assess associations. Overall 68 RCTs were identified. Studies were often chemotherapy-based (n = 41 studies) focusing upon high grade gliomas (46 %) and metastases (41 %) as the top pathologies. Multi-center trials (71 %) were frequent. The overall median CONSORT and Jadad scores were 34.5 (maximum 44) and 2 (maximum 5), respectively; these scores were similar in radiation and chemotherapy-based trials. Major areas of deficiency pertained to allocation concealment, implementation of methods, and blinding whereby less than 20 % of articles fulfilled all criteria. Description of intervention, random sequence generation, and the details regarding recruitment were also deficient; less than 50 % of studies fulfilled all criteria. Description of sample size calculations and blinding improved in later published cohorts. Journal impact factor was significantly associated with higher quality (p = 0.04). Large academic consortia, multi-center designs, ITT analysis, collaboration with biostatisticians, larger sample sizes, and studies with pragmatic objectives were more likely to achieve positive primary outcomes on univariate analysis; none of these variables were significant on multivariate analysis. Deficiencies in the quality of design/reporting of RCTs in neuro-oncology persist. Quality improvement is necessary. Consideration of alternative strategies should be considered.

Treatment of Blood Cholesterol to Reduce Risk for Atherosclerotic Cardiovascular Disease: Grand Rounds Discussion From the Beth Israel Deaconess Medical Center

In November 2013, the American College of Cardiology and the American Heart Association released a clinical practice guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular disease (ASCVD) risk in adults. The recommendation identifies 4 patient groups with strong evidence that the benefits of reduction in ASCVD events from statin therapy exceed adverse events. For these patients, initiating statin therapy of an appropriate intensity to reduce ASCVD risk and minimize adverse effects is recommended. A new risk estimator based on a pooled cohort equation is presented for estimating 10-year ASCVD risk. There is also a recommendation to engage in a clinician-patient discussion before initiating a statin, especially for primary prevention of ASCVD. This paper summarizes a discussion between a cardiologist and an internist about how each clinician would balance these factors and what treatment they would suggest for an individual patient.

From Randomized Controlled Trials of Antidepressant Drugs to the Meta-Analytic Synthesis of Evidence: Methodological Aspects Lead to Discrepant Findings

During the last decade, several meta-analytic studies employing different methodological approaches have had inconsistent conclusions regarding antidepressant efficacy. Herein, we aim to comment on methodological aspects that may have contributed to disparate findings. We initially discuss methodological inconsistencies and limitations related to the conduct of individual antidepressant randomized controlled trials (RCTs), including differences in allocated samples, limitations of psychometric scales, possible explanations for the heightened placebo response rates in antidepressant RCTs across the past two decades as well as the reporting of conflicts of interest. In the second part of this article, we briefly describe the various meta-analyses techniques (e.g., simple random effects meta-analysis and network meta-analysis) and the application of these methods to synthesize evidence related to antidepressant efficacy. Recently published antidepressant metaanalyses often provide discrepant results and similar results often lead to different interpretations. Finally, we propose strategies to improve methodology considering real-world clinical scenarios.

Systematic review of the literature and evidence-based recommendations for antibiotic prophylaxis in trauma: results from an Italian consensus of experts

BACKGROUND

Antibiotic prophylaxis is frequently administered in severe trauma. However, the risk of selecting resistant bacteria, a major issue especially in critical care environments, has not been sufficiently investigated. The aim of the present study was to provide guidelines for antibiotic prophylaxis for four different trauma-related clinical conditions, taking into account the risks of antibiotic-resistant bacteria selection, thus innovating previous guidelines in the field.

METHODS

The MEDLINE database was searched for studies comparing antibiotic prophylaxis to controls (placebo or no antibiotic administration) in four clinical traumatic conditions that were selected on the basis of the traumatic event frequency and/or infection severity. The selected studies focused on the prevention of early ventilator associated pneumonia (VAP) in comatose patients with traumatic brain injury, of meningitis in severe basilar skull fractures, of wound infections in long-bone open fractures. Since no placebo-controlled study was available for deep surgical site-infections prevention in abdominal trauma with enteric contamination, we compared 24-hour and 5-day antibiotic prophylaxis policies. A separate specific research focused on the question of antibiotic-resistant bacteria selection caused by antibiotic prophylaxis, an issue not adequately investigated by the selected studies. Randomised trials, reviews, meta-analyses, observational studies were included. Data extraction was carried out by one author according to a predefined protocol, using an electronic form. The strength of evidence was stratified and recommendations were given according to the Grading of Recommendations Assessment, Development and Evaluation (GRADE) criteria.

RESULTS

Uncertain evidence deserving further studies was found for two-dose antibiotic prophylaxis for early VAP prevention in comatose patients. In the other cases the risk of resistant-bacteria selection caused by antibiotic administration for 48 hours or more, outweighed potential benefits.

CONCLUSIONS

When accounting for antibiotic-resistant bacteria selection we found no evidence in favour of antibiotic prophylaxis lasting two or more days in the studied clinical conditions.

A few good reasons why species-area relationships do not work for parasites

Several studies failed to find strong relationships between the biological and ecological features of a host and the number of parasite species it harbours. In particular, host body size and geographical range are generally only weak predictors of parasite species richness, especially when host phylogeny and sampling effort are taken into account. These results, however, have been recently challenged by a meta-analytic study that suggested a prominent role of host body size and range extent in determining parasite species richness (species-area relationships). Here we argue that, in general, results from meta-analyses should not discourage researchers from investigating the reasons for the lack of clear patterns, thus proposing a few tentative explanations to the fact that species-area relationships are infrequent or at least difficult to be detected in most host-parasite systems. The peculiar structure of host-parasite networks, the enemy release hypothesis, the possible discrepancy between host and parasite ranges, and the evolutionary tendency of parasites towards specialization may explain why the observed patterns often do not fit those predicted by species-area relationships.

Comparative review of the blood pressure-lowering and cardiovascular benefits of telmisartan and perindopril

Hypertension is a major cardiovascular (CV) risk factor, and blood pressure (BP)-lowering treatment substantially reduces the risk. This review compares the available clinical evidence from the BP-lowering and CV-outcome studies of telmisartan and perindopril, which are among the most intensively studied members of their respective classes. The PubMed database was searched for telmisartan and perindopril publications meeting the following criteria: 1) head-to-head comparison trials for BP lowering; and 2) CV-outcome studies (ie, ones with a CV event, mortality, or hospitalization outcome) in patients with CV risk factors but without heart failure. In comparative trials, telmisartan treatment resulted in significantly higher reduction in trough BP and mean ambulatory diastolic BP for the last 8 hours of the dosing interval compared with perindopril. In mainly placebo-controlled CV-outcome studies in patients with hypertension, CV benefits with perindopril were associated with large reductions in BP. There were no CV outcome studies with telmisartan in patients with hypertension. The beyond-BP-lowering CV-protective benefits of telmisartan were demonstrated in the active-controlled ONTARGET (ONgoing Telmisartan Alone and in combination with Ramipril Global Endpoint Trial) trial, which included patients with controlled BP at baseline. In general, the trials discussed in this review reinforce the fact that perindopril and telmisartan are two long-acting antihypertensive drugs that reduce BP over 24 hours, and are the best-evidenced drugs in their class with proven CV protection. It is also clear that the benefits are not a “class effect”, and vary between the different drugs within each class. Hence, the best approach for treatments tailored to individual patient needs should be evidence-based specific drugs, rather than a drug-class recommendation for achieving therapeutic targets.

Burning issues in the meta-analysis of pharmaceutical trials for depression

During the last decade a number of meta-analytic studies have been published and they triggered a debate on the true clinical usefulness of antidepressants. The current article comments on problems within the randomized controlled trials design, the study samples, the psychometric scales, the methods of meta-analysis, the interpretation of the results, and the reporting of conflicts of interest. Although the meta-analyses published so far agree that medication works in severe depression, they question its efficacy in mild cases. However, several methodological issues should be clarified before conclusions are definite. Different methods give different results and similar results seem to entertain a variety of interpretations. In the future it is important to address all of these problems, and to improve methodology on the basis of clinically informed choices. Otherwise, meta-analysis risks alienation from clinical reality and thus risks becoming the 21(st) century psychoanalysis.