False positives in cancer epidemiology

The effect of uncertainty in patient classification on diagnostic performance estimations

Background

The performance of a new diagnostic test is typically evaluated against a comparator which is assumed to correspond closely to some true state of interest. Judgments about the new test’s performance are based on the differences between the outputs of the test and comparator. It is commonly assumed that a small amount of uncertainty in the comparator’s classifications will negligibly affect the measured performance of a diagnostic test.

Methods

Simulated datasets were generated to represent typical diagnostic scenarios. Comparator noise was introduced in the form of random misclassifications, and the effect on the apparent performance of the diagnostic test was determined. An actual dataset from a clinical trial on a new diagnostic test for sepsis was also analyzed.

Results

We demonstrate that as little as 5% misclassification of patients by the comparator can be enough to statistically invalidate performance estimates such as sensitivity, specificity and area under the receiver operating characteristic curve, if this uncertainty is not measured and taken into account. This distortion effect is found to increase non-linearly with comparator uncertainty, under some common diagnostic scenarios. For clinical populations exhibiting high degrees of classification uncertainty, failure to measure and account for this effect will introduce significant risks of drawing false conclusions. The effect of classification uncertainty is magnified further for high performing tests that would otherwise reach near-perfection in diagnostic evaluation trials. A requirement of very high diagnostic performance for clinical adoption, such as a 99% sensitivity, can be rendered nearly unachievable even for a perfect test, if the comparator diagnosis contains even small amounts of uncertainty. This paper and an accompanying online simulation tool demonstrate the effect of classification uncertainty on the apparent performance of tests across a range of typical diagnostic scenarios. Both simulated and real datasets are used to show the degradation of apparent test performance as comparator uncertainty increases.

Conclusions

Overall, a 5% or greater misclassification rate by the comparator can lead to significant underestimation of true test performance. An online simulation tool allows researchers to explore this effect using their own trial parameters (https://imperfect-gold-standard.shinyapps.io/classification-noise/) and the source code is freely available (https://github.com/ksny/Imperfect-Gold-Standard).

The replication crisis in epidemiology: snowball, snow job, or winter solstice?

PURPOSE OF REVIEW

Like a snowball rolling down a steep hill, the most recent crisis over the perceived lack of reproducibility of scientific results has outpaced the evidence of crisis. It has led to new actions and new guidelines that have been rushed to market without plans for evaluation, metrics for success, or due consideration of the potential for unintended consequences.

RECENT FINDINGS

The perception of the crisis is at least partly a snow job, heavily influenced by a small number of centers lavishly funded by a single foundation, with undue and unsupported attention to preregistration as a solution to the perceived crisis. At the same time, the perception of crisis provides an opportunity for introspection. Two studies' estimates of association may differ because of undue attention on null hypothesis statistical testing, because of differences in the distribution of effect modifiers, because of differential susceptibility to threats to validity, or for other reasons. Perhaps the expectation of what reproducible epidemiology ought to look like is more misguided than the practice of epidemiology. We advocate for the idea of "replication and advancement." Studies should not only replicate earlier work, but also improve on it in by enhancing the design or analysis.

SUMMARY

Abandoning blind reliance on null hypothesis significance testing for statistical inference, finding consensus on when pre-registration of non-randomized study protocols has merit, and focusing on replication and advance are the most certain ways to emerge from this solstice for the better.

On the International Agency for Research on Cancer classification of glyphosate as a probable human carcinogen

The recent classification by International Agency for Research on Cancer (IARC) of the herbicide glyphosate as a probable human carcinogen has generated considerable discussion. The classification is at variance with evaluations of the carcinogenic potential of glyphosate by several national and international regulatory bodies. The basis for the IARC classification is examined under the assumptions that the IARC criteria are reasonable and that the body of scientific studies determined by IARC staff to be relevant to the evaluation of glyphosate by the Monograph Working Group is sufficiently complete. It is shown that the classification of glyphosate as a probable human carcinogen was the result of a flawed and incomplete summary of the experimental evidence evaluated by the Working Group. Rational and effective cancer prevention activities depend on scientifically sound and unbiased assessments of the carcinogenic potential of suspected agents. Implications of the erroneous classification of glyphosate with respect to the IARC Monograph Working Group deliberative process are discussed.

Diet, body size, physical activity and risk of prostate cancer: An umbrella review of the evidence

The existing literature on the relationship between diet, body size, physical activity and prostate cancer risk was summarised by the World Cancer Research Fund Continuous Update Project (CUP). An evaluation of the robustness of this evidence is required to help inform public health policy. The robustness of this evidence was evaluated using several criteria addressing evidence strength and validity, including the statistical significance of the random effects summary estimate and of the largest study in a meta-analysis, number of prostate cancer cases, between-study heterogeneity, 95% prediction intervals, small-study effects bias, excess significance bias and sensitivity analyses with credibility ceilings. A total of 248 meta-analyses were extracted from the CUP, which studied associations of 23 foods, 31 nutrients, eight indices of body size and three indices of physical activity with risk of total prostate cancer development, mortality or cancer development by stage and grade. Of the 176 meta-analyses using a continuous scale to measure the exposures, no association presented strong evidence by satisfying all the aforementioned criteria. Only the association of height with total prostate cancer incidence and mortality presented highly suggestive evidence with a 4% higher risk per 5 cm greater height (95% confidence interval, 1.03, 1.05). Associations for body mass index, weight, height, dietary calcium and spirits intake were supported by suggestive evidence. Overall, the association of diet, body size, physical activity and prostate cancer has been extensively studied, but no association was graded with strong evidence.

The Activation of ERK1/2 and JNK MAPK Signaling by Insulin/IGF-1 Is Responsible for the Development of Colon Cancer with Type 2 Diabetes Mellitus

Previous studies showed that type 2 diabetes mellitus (T2DM) is linked to increased risk of developing colon cancer. Insulin and insulin-like growth factor 1 (IGF-1) are increased in patients with T2DM. The increased insulin and IGF-1 may be responsible for the developing of colon cancer. In this study, we investigated the effects and mechanisms of insulin and IGF-1 in colon cancer development in vitro and in vivo. Insulin and IGF-1 alone or together elevated proliferation and reduced apoptosis in colon cancer MC38 cells. Meanwhile, insulin and IGF-1 promoted the phosphorylation of extracellular-signal regulated kinase 1/2 (ERK1/2) and c-Jun N-terminal kinase (JNK). Treatment with ERK1/2 or JNK inhibitor in the presence of insulin and IGF-1 significantly decreased B-cell lymphoma 2 (Bcl-2) and increased Bcl-2-associated X protein (Bax) expression and finally increased apoptosis and inhibited the proliferation. Accelerative colon tumor growth was found in a mouse model of T2DM with db/db mice which got high level of endogenous insulin and IGF-1. Furthermore, the inhibition of ERK1/2 or JNK suppressed the development of colon tumor in vivo. These results suggest that the activation of ERK1/2 and JNK signaling by insulin and IGF-1, at least in part, is responsible for the development of colon cancer with T2DM.

IARC monographs: 40 years of evaluating carcinogenic hazards to humans

BACKGROUND

Recently, the International Agency for Research on Cancer (IARC) Programme for the Evaluation of Carcinogenic Risks to Humans has been criticized for several of its evaluations, and also for the approach used to perform these evaluations. Some critics have claimed that failures of IARC Working Groups to recognize study weaknesses and biases of Working Group members have led to inappropriate classification of a number of agents as carcinogenic to humans.

OBJECTIVES

The authors of this Commentary are scientists from various disciplines relevant to the identification and hazard evaluation of human carcinogens. We examined criticisms of the IARC classification process to determine the validity of these concerns. Here, we present the results of that examination, review the history of IARC evaluations, and describe how the IARC evaluations are performed.

DISCUSSION

We concluded that these recent criticisms are unconvincing. The procedures employed by IARC to assemble Working Groups of scientists from the various disciplines and the techniques followed to review the literature and perform hazard assessment of various agents provide a balanced evaluation and an appropriate indication of the weight of the evidence. Some disagreement by individual scientists to some evaluations is not evidence of process failure. The review process has been modified over time and will undoubtedly be altered in the future to improve the process. Any process can in theory be improved, and we would support continued review and improvement of the IARC processes. This does not mean, however, that the current procedures are flawed.

CONCLUSIONS

The IARC Monographs have made, and continue to make, major contributions to the scientific underpinning for societal actions to improve the public's health.

Issues of methods and interpretation in the National Cancer Institute formaldehyde cohort study

In 2004, the International Agency for Research on Cancer (IARC) reclassified formaldehyde (FA) from a probable (Group 2A) to a known human carcinogen (Group 1) citing results for nasopharyngeal cancer (NPC) mortality from the follow-up through 1994 of the National Cancer Institute formaldehyde cohort study. To the contrary, in 2012, the Committee for Risk Assessment of the European Chemicals Agency disagreed with the proposal to classify FA as a known human carcinogen (Carc. 1A), proposing a lower but still protective category, namely as a substance which is presumed to have carcinogenic potential for humans (Carc. 1B). Thus, U.S. and European regulatory agencies currently disagree about the potential human carcinogenicity of FA. In 2013, the National Cancer Institute reported results from their follow-up through 2004 of the formaldehyde cohort and concluded that the results continue to suggest a link between FA exposure and NPC. We discuss in this commentary why we believe that this interpretation is neither consistent with the available data from the most recent update of the National Cancer Institute cohort study nor with other research findings from that cohort, other large cohort studies and the series of publications by some of the current authors, including an independent study of one of the National Cancer Institute's study plants. Another serious concern relates to the incorrectness of the data from the follow-up through 1994 of the National Cancer Institute study stemming from incomplete mortality ascertainment. While these data were corrected by the National Cancer Institute in subsequent supplemental publications, incorrect data from the original publications have been cited extensively in recent causal evaluations of FA, including IARC. We conclude that the NCI publications that contain incorrect data from the incomplete 1994 mortality follow-up should be retracted entirely or corrected via published errata in the corresponding journals, and efforts should be made to re-analyze data from the 2004 follow-up of the NCI cohort study.