A psychometric experiment in causal inference to estimate evidential weights used by epidemiologists

The reporting of p values, confidence intervals and statistical significance in Preventive Veterinary Medicine (1997-2017)

Background

Despite much discussion in the epidemiologic literature surrounding the use of null hypothesis significance testing (NHST) for inferences, the reporting practices of veterinary researchers have not been examined. We conducted a survey of articles published in Preventive Veterinary Medicine, a leading veterinary epidemiology journal, aimed at (a) estimating the frequency of reporting p values, confidence intervals and statistical significance between 1997 and 2017, (b) determining whether this varies by article section and (c) determining whether this varies over time.

Methods

We used systematic cluster sampling to select 985 original research articles from issues published in March, June, September and December of each year of the study period. Using the survey data analysis menu in Stata, we estimated overall and yearly proportions of article sections (abstracts, results-texts, results-tables and discussions) reporting p values, confidence intervals and statistical significance. Additionally, we estimated the proportion of p values less than 0.05 reported in each section, the proportion of article sections in which p values were reported as inequalities, and the proportion of article sections in which confidence intervals were interpreted as if they were significance tests. Finally, we used Generalised Estimating Equations to estimate prevalence odds ratios and 95% confidence intervals, comparing the occurrence of each of the above-mentioned reporting elements in one article section relative to another.

Results

Over the 20-year period, for every 100 published manuscripts, 31 abstracts (95% CI [28–35]), 65 results-texts (95% CI [61–68]), 23 sets of results-tables (95% CI [20–27]) and 59 discussion sections (95% CI [56–63]) reported statistical significance at least once. Only in the case of results-tables, were the numbers reporting p values (48; 95% CI [44–51]), and confidence intervals (44; 95% CI [41–48]) higher than those reporting statistical significance. We also found that a substantial proportion of p values were reported as inequalities and most were less than 0.05. The odds of a p value being less than 0.05 (OR = 4.5; 95% CI [2.3–9.0]) or being reported as an inequality (OR = 3.2; 95% CI [1.3–7.6]) was higher in the abstracts than in the results-texts. Additionally, when confidence intervals were interpreted, on most occasions they were used as surrogates for significance tests. Overall, no time trends in reporting were observed for any of the three reporting elements over the study period.

Conclusions

Despite the availability of superior approaches to statistical inference and abundant criticism of its use in the epidemiologic literature, NHST is substantially the most common means of inference in articles published in Preventive Veterinary Medicine. This pattern has not changed substantially between 1997 and 2017.

The replication crisis in epidemiology: snowball, snow job, or winter solstice?

PURPOSE OF REVIEW

Like a snowball rolling down a steep hill, the most recent crisis over the perceived lack of reproducibility of scientific results has outpaced the evidence of crisis. It has led to new actions and new guidelines that have been rushed to market without plans for evaluation, metrics for success, or due consideration of the potential for unintended consequences.

RECENT FINDINGS

The perception of the crisis is at least partly a snow job, heavily influenced by a small number of centers lavishly funded by a single foundation, with undue and unsupported attention to preregistration as a solution to the perceived crisis. At the same time, the perception of crisis provides an opportunity for introspection. Two studies' estimates of association may differ because of undue attention on null hypothesis statistical testing, because of differences in the distribution of effect modifiers, because of differential susceptibility to threats to validity, or for other reasons. Perhaps the expectation of what reproducible epidemiology ought to look like is more misguided than the practice of epidemiology. We advocate for the idea of "replication and advancement." Studies should not only replicate earlier work, but also improve on it in by enhancing the design or analysis.

SUMMARY

Abandoning blind reliance on null hypothesis significance testing for statistical inference, finding consensus on when pre-registration of non-randomized study protocols has merit, and focusing on replication and advance are the most certain ways to emerge from this solstice for the better.

Design, analysis, and conclusions: Telling a consistent causal story

Pharmacy educators are dedicated to providing the best education to student pharmacists and post-graduate trainees. This involves developing and evaluating new educational approaches or activities, as well as working to understand better other factors that may affect student and program outcomes. Although we may not realize it, the idea of causality, or the presence of a cause-and-effect relationship, is embedded in much of what we do when engaging in pharmacy educational scholarship. Saying that an educational intervention is effective at improving test scores implies that the intervention caused the increase. Perhaps more subtly, a cause-and-effect relationship is implied when identifying poor interviewing skills as a factor that reduces the likelihood that a student will obtain a pharmacy practice residency. The extent to which causal claims can be made depends on appropriate study design and analysis. Similarly, the language used to present these claims is related to study design and analysis. Unfortunately, study designs used when evaluating educational approaches do not always align with the language used to report the results. This review provides a brief overview of current thinking related to causal inference. The role of study design and analysis in causal inference is also discussed along with recommendations for study design and analysis to facilitate making appropriate causal claims. After reading this article, readers should be able to select a study design that best aligns with their particular research question and report their findings in ways that are faithful to the strengths of the study design.

Molecular epidemiology of acute leukemia in children: causal model, interaction of three factors-susceptibility, environmental exposure and vulnerability period

Acute leukemias have a huge morphological, cytogenetic and molecular heterogeneity and genetic polymorphisms associated with susceptibility. Every leukemia presents causal factors associated with the development of the disease. Particularly, when three factors are present, they result in the development of acute leukemia. These phenomena are susceptibility, environmental exposure and a period that, for this model, has been called the period of vulnerability. This framework shows how the concepts of molecular epidemiology have established a reference from which it is more feasible to identify the environmental factors associated with the development of leukemia in children. Subsequently, the arguments show that only susceptible children are likely to develop leukemia once exposed to an environmental factor. For additional exposure, if the child is not susceptible to leukemia, the disease does not develop. In addition, this exposure should occur during a time window when hematopoietic cells and their environment are more vulnerable to such interaction, causing the development of leukemia. This model seeks to predict the time when the leukemia develops and attempts to give a context in which the causality of childhood leukemia should be studied. This information can influence and reduce the risk of a child developing leukemia.

Toxicology research for precautionary decision-making and the role of Human & Experimental Toxicology

A key aim of toxicology is the prevention of adverse effects due to toxic hazards. Therefore, the dissemination of toxicology research findings must confront two important challenges: one being the lack of information on the vast majority of potentially toxic industrial chemicals and the other being the strict criteria for scientific proof usually required for decision-making in regard to prevention. The present study ascertains the coverage of environmental chemicals in four volumes of Human & Experimental Toxicology and the presentation and interpretation of research findings in published articles. Links in SciFinder showed that the 530 articles published in four selected volumes between 1984 and 2014 primarily dealt with metals (126 links) and other toxicants that have received substantial attention in the past. Thirteen compounds identified by US authorities in 2006 as high-priority substances, for which toxicology documentation is badly needed, were not covered in the journal issues at all. When reviewing published articles, reliance on p values was standard, and non-significant findings were often called 'negative.' This tradition may contribute to the perceived need to extend existing research on toxic hazards that have already been well characterized. Several sources of bias towards the null hypothesis can affect toxicology research, but are generally not considered, thus adding to the current inclination to avoid false positive findings. In this regard, toxicology is particularly prone to bias because of the known paucity of false positives and, in particular, the existence of a vast number of toxic hazards which by default are considered innocuous due to lack of documentation. The Precautionary Principle could inspire decision-making on the basis of incomplete documentation and should stimulate a change in toxicology traditions and in toxicology research publication.

Hume, Mill, Hill, and the sui generis epidemiologic approach to causal inference

The epidemiologic approach to causal inference (i.e., Hill's viewpoints) consists of evaluating potential causes from the following 2, noncumulative angles: 1) established results from comparative, observational, or experimental epidemiologic studies; and 2) reviews of nonepidemiologic evidence. It does not involve statements of statistical significance. The philosophical roots of Hill's viewpoints are unknown. Superficially, they seem to descend from the ideas of Hume and Mill. Hill's viewpoints, however, use a different kind of evidence and have different purposes than do Hume's rules or Mill's system of logic. In a nutshell, Hume ignores comparative evidence central to Hill's viewpoints. Mill's logic disqualifies as invalid nonexperimental evidence, which forms the bulk of epidemiologic findings reviewed from Hill's viewpoints. The approaches by Hume and Mill cannot corroborate successful implementations of Hill's viewpoints. Besides Hume and Mill, the epidemiologic literature is clueless about a plausible, pre-1965 philosophical origin of Hill's viewpoints. Thus, Hill's viewpoints may be philosophically novel, sui generis, still waiting to be validated and justified.

How physicians allocate causation: a scenario study with factorial design

BACKGROUND

Causation is a complex concept but important in suspected work-related disease. Physicians routinely make initial assessments of causation as part of their work, but the factors contributing to these assessments are not well understood.

AIMS

To determine which factors influence a family physician's assessment of causation when seeing patients with suspected work-related injury or illness.

METHODS

Four groups of family physicians with differing levels of prior reporting (zero, low, medium, high) to the Workers Compensation Board received a questionnaire including four randomly allocated case scenarios. For each scenario there were four versions with either strong or weak causal features suggesting work or non-work factors were important causes or contributors. Responses to questions were made on a series of visual analogue scales.

RESULTS

The nature of the condition and scenario type (i.e. strength of the causal information about workplace and non-workplace factors) were associated with the physicians' opinion on work-relatedness. Understanding the nature of the patient's work, the timing of symptoms and the patients' opinion about work-relatedness were viewed by the physicians as important. A decision that a condition was not work related was influenced primarily by the strength of potential causes outside work. Prior reporting history of the physician was not associated with opinions on work-relatedness, nor the factors considered in reaching this decision.

CONCLUSIONS

The characteristics of the case scenario were more important in determining a physician's opinion about work-relatedness than the characteristics of the physician.

When Work is Related to Disease, What Establishes Evidence for a Causal Relation?

Establishing a causal relationship between factors at work and disease is difficult for occupational physicians and researchers. This paper seeks to provide arguments for the judgement of evidence of causality in observational studies that relate work factors to disease. I derived criteria for the judgement of evidence of causality from the following sources: the criteria list of Hill, the approach by Rothman, the methods used by International Agency for Research on Cancer (IARC), and methods used by epidemiologists. The criteria are applied to two cases of putative occupational diseases; breast cancer caused by shift work and aerotoxic syndrome. Only three of the Hill criteria can be applied to an actual study. Rothman stresses the importance of confounding and alternative explanations than the putative cause. IARC closely follows Hill, but they also incorporate other than epidemiological evidence. Applied to shift work and breast cancer, these results have found moderate evidence for a causal relationship, but applied to the aerotoxic syndrome, there is an absence of evidence of causality. There are no ready to use algorithms for judgement of evidence of causality. Criteria from different sources lead to similar results and can make a conclusion of causality more or less likely.

Heuristic thinking and inference from observational epidemiology

Epidemiologic research is an exercise in measurement. Observational epidemiologic results usually include a point estimate, a measure of random error such as a frequentist confidence interval, and a qualitative discussion of study limitations. Without randomization of study subjects to exposure groups, inference from study results requires an educated guess about the strength of the systematic errors compared with the strength of the exposure effects. Although quantitative methods to make these educated guesses exist, the conventional approach is qualitative, which reduces the educated guessing to a problem of reasoning under uncertainty. In circumstances such as these, humans predictably reason poorly. Heuristics and resulting biases that simplify the judgmental tasks tend to underestimate the systematic error, underestimate the uncertainty, and focus the inference on the study's specific evidence while excluding countervailing external information. Common warnings to interpret results with trepidation are an ineffective solution. The methods that quantify systematic error and uncertainty challenge the analyst to specify the alternative explanations for associations that are otherwise too readily judged causal.

Causal criteria and counterfactuals; nothing more (or less) than scientific common sense

Two persistent myths in epidemiology are that we can use a list of "causal criteria" to provide an algorithmic approach to inferring causation and that a modern "counterfactual model" can assist in the same endeavor. We argue that these are neither criteria nor a model, but that lists of causal considerations and formalizations of the counterfactual definition of causation are nevertheless useful tools for promoting scientific thinking. They set us on the path to the common sense of scientific inquiry, including testing hypotheses (really putting them to a test, not just calculating simplistic statistics), responding to the Duhem-Quine problem, and avoiding many common errors. Austin Bradford Hill's famous considerations are thus both over-interpreted by those who would use them as criteria and under-appreciated by those who dismiss them as flawed. Similarly, formalizations of counterfactuals are under-appreciated as lessons in basic scientific thinking. The need for lessons in scientific common sense is great in epidemiology, which is taught largely as an engineering discipline and practiced largely as technical tasks, making attention to core principles of scientific inquiry woefully rare.

Weight of evidence: a review of concept and methods

"Weight of evidence" (WOE) is a common term in the published scientific and policy-making literature, most often seen in the context of risk assessment (RA). Its definition, however, is unclear. A systematic review of the scientific literature was undertaken to characterize the concept. For the years 1994 through 2004, PubMed was searched for publications in which "weight of evidence" appeared in the abstract and/or title. Of the 276 papers that met these criteria, 92 were selected for review: 71 papers published in 2003 and 2004 (WOE appeared in abstract/title) and 21 from 1994 through 2002 (WOE appeared in title). WOE has three characteristic uses in this literature: (1) metaphorical, where WOE refers to a collection of studies or to an unspecified methodological approach; (2) methodological, where WOE points to established interpretative methodologies (e.g., systematic narrative review, meta-analysis, causal criteria, and/or quality criteria for toxicological studies) or where WOE means that "all" rather than some subset of the evidence is examined, or rarely, where WOE points to methods using quantitative weights for evidence; and (3) theoretical, where WOE serves as a label for a conceptual framework. Several problems are identified: the frequent lack of definition of the term "weight of evidence," multiple uses of the term and a lack of consensus about its meaning, and the many different kinds of weights, both qualitative and quantitative, which can be used in RA. A practical recommendation emerges: the WOE concept and its associated methods should be fully described when used. A research agenda should examine the advantages of quantitative versus qualitative weighting schemes, how best to improve existing methods, and how best to combine those methods (e.g., epidemiology's causal criteria with toxicology's quality criteria).

Associations Between Periodontal Disease and Systemic Disease: Evaluating the Strength of the Evidence

Much work has been published on the association between periodontal disease and systemic disease, including original reports, narrative reviews, systematic reviews, and meta-analyses. Based on the existent work, one can assign an evidence level and grade, using standard evidence-based criteria, to the data available in the four major categories of medical outcomes studied: cardiovascular/cerebrovascular, pregnancy, pulmonary, and diabetes. We discuss methodologic and conceptual problems in the study of oral-systemic associations, focusing as an example on the association between periodontal disease and cardiovascular/cerebrovascular disease. We argue that the hierarchical ranking of studies by levels of evidence may be misleading. In particular, while randomized controlled trials (RCTs) are needed to determine the efficacy of periodontal treatment to reduce the risk of cardiovascular events, they may be of limited value in determining the etiologic role of periodontal disease on coronary heart disease and stroke. We discuss limitations of RCTs as well as the limitations of currently available data from epidemiologic studies, including study design and confounding and misclassification errors. We conclude that well-designed observational studies into the associations between periodontal disease and systemic disease need to remain an integral component of future research efforts in order to fully understand such associations.

Assisted reproductive technologies and the risk of birth defects—a systematic review

BACKGROUND

The risk of birth defects in infants born following assisted reproductive technology (ART) treatment is a controversial question. Most publications examining the prevalence of birth defects in ICSI and IVF infants compared to spontaneously conceived infants have serious methodological limitations; despite this, most researchers have concluded that there is no increased risk.

METHODS

We carried out a systematic review to identify all papers published by March 2003 with data relating to the prevalence of birth defects in infants conceived following IVF and/or ICSI compared with spontaneously conceived infants. Independent expert reviewers used criteria defined a priori to determine whether studies were suitable for inclusion in a meta-analysis. Fixed effects meta-analysis was performed for all studies and reviewer-selected studies.

RESULTS

Twenty-five studies were identified for review. Two-thirds of these showed a 25% or greater increased risk of birth defects in ART infants. The results of meta-analyses of the seven reviewer-selected studies and of all 25 studies suggest a statistically significant 30-40% increased risk of birth defects associated with ART.

CONCLUSIONS

Pooled results from all suitable published studies suggest that children born following ART are at increased risk of birth defects compared with spontaneous conceptions. This information should be made available to couples seeking ART treatment.

Evidence based practice in population health: a regional survey to inform workforce development and organisational change

STUDY OBJECTIVE

To assess current capacity to implement evidence based practice (EBP) in population health.

DESIGN

Postal survey of a regional population health workforce in Sydney, Australia.

SETTING

Division of Population Health, South Western Sydney Area Health Service.

PARTICIPANTS

104 population health staff (response rate: 73%).

MAIN RESULTS

In the sample of regional population health practitioners, views about the current promotion of EBP were positive. Non-medical respondents with less that Masters degree were more likely to report "high self assessed need" to increase their capacity in EBP (p = 0.022). Confidence in understanding of EBP terminology was not associated with seniority but with highest level of education reached (p<0.000) and having medical qualifications (p<0.000). Occupational category was not associated with respondents' self assessed "need for evidence", "need for EBP skills" or "need to increase their capacity in EBP" in their current position. The proportion of participants "strongly" supporting implementation of a colorectal cancer screening programme whose benefit was expressed as relative risk reduction was greater than that so supporting a programme whose benefit was expressed as number needed to screen (p = 0.008). Most respondents referred to their immediate managers when seeking support for EBP.

CONCLUSIONS

The findings provide a quantitative baseline for capacity building through workplace programmes. Managerial commitment has been increased and performance development is now underway.

Association chain graphs: modelling etiological pathways

Multiple time-dynamic and interrelated risk factors are usually involved in the complex etiology of disorders. This paper presents a strategy to explore and display visually the relative importance of different association pathways for the onset of disorder over time. The approach is based on graphical chain models, a tool that is powerful but still under-utilized in most fields. Usually, the results of these models are displayed using directed acyclic graphs (DAGs). These draw an edge between a pair of variables whenever the assumption of conditional independence given variables on an earlier or equal temporal footing is violated to a statistically significant extent. In the present paper, the graphs are modified in that confidence intervals for the strengths of associations (statistical main effects) are visualized. These new graphs are called association chain graphs (ACGs). Statistical interactions cause 'edges' between the respective variables within the DAG framework (because the assumption of conditional independence is violated). In contrast they are represented as separate graphs within the subsample where the different association chains may work within the ACG framework. With this new type of graph, more specific information can be displayed whenever the data are essentially described only with statistical main- and two-way interaction effects.