Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates

The most widely used task functional magnetic resonance imaging (fMRI) analyses use parametric statistical methods that depend on a variety of assumptions. In this work, we use real resting-state data and a total of 3 million random task group analyses to compute empirical familywise error rates for the fMRI software packages SPM, FSL, and AFNI, as well as a nonparametric permutation method. For a nominal familywise error rate of 5%, the parametric statistical methods are shown to be conservative for voxelwise inference and invalid for clusterwise inference. Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape. By comparison, the nonparametric permutation test is found to produce nominal results for voxelwise as well as clusterwise inference. These findings speak to the need of validating the statistical methods being used in the field of neuroimaging.

Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability

An academic scientist's professional success depends on publishing. Publishing norms emphasize novel, positive results. As such, disciplinary incentives encourage design, analysis, and reporting decisions that elicit positive results and ignore negative results. Prior reports demonstrate how these incentives inflate the rate of false effects in published science. When incentives favor novelty over replication, false results persist in the literature unchallenged, reducing efficiency in knowledge accumulation. Previous suggestions to address this problem are unlikely to be effective. For example, a journal of negative results publishes otherwise unpublishable reports. This enshrines the low status of the journal and its content. The persistence of false findings can be meliorated with strategies that make the fundamental but abstract accuracy motive-getting it right-competitive with the more tangible and concrete incentive-getting it published. This article develops strategies for improving scientific practices and knowledge accumulation that account for ordinary human motivations and biases.

The harmonic mean p-value for combining dependent tests

Analysis of "big data" frequently involves statistical comparison of millions of competing hypotheses to discover hidden processes underlying observed patterns of data, for example, in the search for genetic determinants of disease in genome-wide association studies (GWAS). Controlling the familywise error rate (FWER) is considered the strongest protection against false positives but makes it difficult to reach the multiple testing-corrected significance threshold. Here, I introduce the harmonic mean p-value (HMP), which controls the FWER while greatly improving statistical power by combining dependent tests using generalized central limit theorem. I show that the HMP effortlessly combines information to detect statistically significant signals among groups of individually nonsignificant hypotheses in examples of a human GWAS for neuroticism and a joint human-pathogen GWAS for hepatitis C viral load. The HMP simultaneously tests all ways to group hypotheses, allowing the smallest groups of hypotheses that retain significance to be sought. The power of the HMP to detect significant hypothesis groups is greater than the power of the Benjamini-Hochberg procedure to detect significant hypotheses, although the latter only controls the weaker false discovery rate (FDR). The HMP has broad implications for the analysis of large datasets, because it enhances the potential for scientific discovery.

Implementation errors in the GingerALE Software: Description and recommendations

Neuroscience imaging is a burgeoning, highly sophisticated field the growth of which has been fostered by grant-funded, freely distributed software libraries that perform voxel-wise analyses in anatomically standardized three-dimensional space on multi-subject, whole-brain, primary datasets. Despite the ongoing advances made using these non-commercial computational tools, the replicability of individual studies is an acknowledged limitation. Coordinate-based meta-analysis offers a practical solution to this limitation and, consequently, plays an important role in filtering and consolidating the enormous corpus of functional and structural neuroimaging results reported in the peer-reviewed literature. In both primary data and meta-analytic neuroimaging analyses, correction for multiple comparisons is a complex but critical step for ensuring statistical rigor. Reports of errors in multiple-comparison corrections in primary-data analyses have recently appeared. Here, we report two such errors in GingerALE, a widely used, US National Institutes of Health (NIH)-funded, freely distributed software package for coordinate-based meta-analysis. These errors have given rise to published reports with more liberal statistical inferences than were specified by the authors. The intent of this technical report is threefold. First, we inform authors who used GingerALE of these errors so that they can take appropriate actions including re-analyses and corrective publications. Second, we seek to exemplify and promote an open approach to error management. Third, we discuss the implications of these and similar errors in a scientific environment dependent on third-party software. Hum Brain Mapp 38:7-11, 2017. © 2016 Wiley Periodicals, Inc.

A Nonlinear Simulation Framework Supports Adjusting for Age When Analyzing BrainAGE

Several imaging modalities, including T1-weighted structural imaging, diffusion tensor imaging, and functional MRI can show chronological age related changes. Employing machine learning algorithms, an individual's imaging data can predict their age with reasonable accuracy. While details vary according to modality, the general strategy is to: (1) extract image-related features, (2) build a model on a training set that uses those features to predict an individual's age, (3) validate the model on a test dataset, producing a predicted age for each individual, (4) define the "Brain Age Gap Estimate" (BrainAGE) as the difference between an individual's predicted age and his/her chronological age, (5) estimate the relationship between BrainAGE and other variables of interest, and (6) make inferences about those variables and accelerated or delayed brain aging. For example, a group of individuals with overall positive BrainAGE may show signs of accelerated aging in other variables as well. There is inevitably an overestimation of the age of younger individuals and an underestimation of the age of older individuals due to "regression to the mean." The correlation between chronological age and BrainAGE may significantly impact the relationship between BrainAGE and other variables of interest when they are also related to age. In this study, we examine the detectability of variable effects under different assumptions. We use empirical results from two separate datasets [training = 475 healthy volunteers, aged 18-60 years (259 female); testing = 489 participants including people with mood/anxiety, substance use, eating disorders and healthy controls, aged 18-56 years (312 female)] to inform simulation parameter selection. Outcomes in simulated and empirical data strongly support the proposal that models incorporating BrainAGE should include chronological age as a covariate. We propose either including age as a covariate in step 5 of the above framework, or employing a multistep procedure where age is regressed on BrainAGE prior to step 5, producing BrainAGE Residualized (BrainAGER) scores.

False discovery rates of protein identifications: a strike against the two-peptide rule

Most proteomics studies attempt to maximize the number of peptide identifications and subsequently infer proteins containing two or more peptides as reliable protein identifications. In this study, we evaluate the effect of this "two-peptide" rule on protein identifications, using multiple search tools and data sets. Contrary to the intuition, the "two-peptide" rule reduces the number of protein identifications in the target database more significantly than in the decoy database and results in increased false discovery rates, compared to the case when single-hit proteins are not discarded. We therefore recommend that the "two-peptide" rule should be abandoned, and instead, protein identifications should be subject to the estimation of error rates, as is the case with peptide identifications. We further extend the generating function approach (originally proposed for evaluating matches between a peptide and a single spectrum) to evaluating matches between a protein and an entire spectral data set.

Detecting false-positive signals in exome sequencing

Disease gene discovery has been transformed by affordable sequencing of exomes and genomes. Identification of disease-causing mutations requires sifting through a large number of sequence variants. A subset of the variants are unlikely to be good candidates for disease causation based on one or more of the following criteria: (1) being located in genomic regions known to be highly polymorphic, (2) having characteristics suggesting assembly misalignment, and/or (3) being labeled as variants based on misleading reference genome information. We analyzed exome sequence data from 118 individuals in 29 families seen in the NIH Undiagnosed Diseases Program (UDP) to create lists of variants and genes with these characteristics. Specifically, we identified several groups of genes that are candidates for provisional exclusion during exome analysis: 23,389 positions with excess heterozygosity suggestive of alignment errors and 1,009 positions in which the hg18 human genome reference sequence appeared to contain a minor allele. Exclusion of such variants, which we provide in supplemental lists, will likely enhance identification of disease-causing mutations using exome sequence data.

Multibatch TMT Reveals False Positives, Batch Effects and Missing Values

Multiplexing strategies for large-scale proteomic analyses have become increasingly prevalent, tandem mass tags (TMT) in particular. Here we used a large iPSC proteomic experiment with twenty-four 10-plex TMT batches to evaluate the effect of integrating multiple TMT batches within a single analysis. We identified a significant inflation rate of protein missing values as multiple batches are integrated and show that this pattern is aggravated at the peptide level. We also show that without normalization strategies to address the batch effects, the high precision of quantitation within a single multiplexed TMT batch is not reproduced when data from multiple TMT batches are integrated.Further, the incidence of false positives was studied by using Y chromosome peptides as an internal control. The iPSC lines quantified in this data set were derived from both male and female donors, hence the peptides mapped to the Y chromosome should be absent from female lines. Nonetheless, these Y chromosome-specific peptides were consistently detected in the female channels of all TMT batches. We then used the same Y chromosome specific peptides to quantify the level of ion coisolation as well as the effect of primary and secondary reporter ion interference. These results were used to propose solutions to mitigate the limitations of multi-batch TMT analyses. We confirm that including a common reference line in every batch increases precision by facilitating normalization across the batches and we propose experimental designs that minimize the effect of cross population reporter ion interference.

Bactericidal antibody against a representative epidemiological meningococcal serogroup B panel confirms that MATS underestimates 4CMenB vaccine strain coverage

BACKGROUND

4CMenB (Bexsero), a vaccine developed against invasive meningococcal disease caused by capsular group B strains (MenB), was recently licensed for use by the European Medicines Agency. Assessment of 4CMenB strain coverage in specific epidemiologic settings is of primary importance to predict vaccination impact on the burden of disease. The Meningococcal Antigen Typing System (MATS) was developed to predict 4CMenB strain coverage, using serum bactericidal antibody assay with human complement (hSBA) data from a diverse panel of strains not representative of any specific epidemiology.

OBJECTIVE

To experimentally validate the accuracy of MATS-based predictions against strains representative of a specific epidemiologic setting.

METHODS AND RESULTS

We used a stratified sampling method to identify a representative sample from all MenB disease isolates collected from England and Wales in 2007-2008, tested the strains in the hSBA assay with pooled sera from infant and adolescent vaccinees, and compared these results with MATS. MATS predictions and hSBA results were significantly associated (P=0.022). MATS predicted coverage of 70% (95% CI, 55-85%) was largely confirmed by 88% killing in the hSBA (95% CI, 72-95%). MATS had 78% accuracy and 96% positive predictive value against hSBA.

CONCLUSION

MATS is a conservative predictor of strain coverage by the 4CMenB vaccine in infants and adolescents.

The Long Way From α-Error Control to Validity Proper: Problems With a Short-Sighted False-Positive Debate

Several influential publications have sensitized the community of behavioral scientists to the dangers of inflated effects and false-positive errors leading to the unwarranted publication of nonreplicable findings. This issue has been related to prominent cases of data fabrication and survey results pointing to bad practices in empirical science. Although we concur with the motives behind these critical arguments, we note that an isolated debate of false positives may itself be misleading and counter-productive. Instead, we argue that, given the current state of affairs in behavioral science, false negatives often constitute a more serious problem. Referring to Wason's (1960) seminal work on inductive reasoning, we show that the failure to assertively generate and test alternative hypotheses can lead to dramatic theoretical mistakes, which cannot be corrected by any kind of rigor applied to statistical tests of the focal hypotheses. We conclude that a scientific culture rewarding strong inference (Platt, 1964) is more likely to see progress than a culture preoccupied with tightening its standards for the mere publication of original findings.

Diagnostic Issues and Controversies in DSM-5: Return of the False Positives Problem

The fifth revision of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) was the most controversial in the manual's history. This review selectively surveys some of the most important changes in DSM-5, including structural/organizational changes, modifications of diagnostic criteria, and newly introduced categories. It analyzes why these changes led to such heated controversies, which included objections to the revision's process, its goals, and the content of altered criteria and new categories. The central focus is on disputes concerning the false positives problem of setting a valid boundary between disorder and normal variation. Finally, this review highlights key problems and issues that currently remain unresolved and need to be addressed in the future, including systematically identifying false positive weaknesses in criteria, distinguishing risk from disorder, including context in diagnostic criteria, clarifying how to handle fuzzy boundaries, and improving the guidelines for "other specified" diagnosis.

Inhibition of IgE binding to cross-reactive carbohydrate determinants enhances diagnostic selectivity

Background

Allergy diagnosis by determination of allergen-specific IgE is complicated by clinically irrelevant IgE, of which the most prominent example is IgE against cross-reactive carbohydrate determinants (CCDs) that occur on allergens from plants and insects. Therefore, CCDs cause numerous false-positive results. Inhibition of CCDs has been proposed as a remedy, but has not yet found its way into the routine diagnostic laboratory. We sought to provide a simple and affordable procedure to overcome the CCD problem.

Methods

Serum samples from allergic patients were analysed for allergen-specific IgEs by different commercial tests (from Mediwiss, Phadia and Siemens) with and without a semisynthetic CCD blocker with minimized potential for nonspecific interactions that was prepared from purified bromelain glycopeptides and human serum albumin.

Results

Twenty two per cent of about 6000 serum samples reacted with CCD reporter proteins. The incidence of anti-CCD IgE reached 35% in the teenage group. In patients with anti-CCD IgE, application of the CCD blocker led to a clear reduction in read-out values, often below the threshold level. A much better correlation between laboratory results and anamnesis and skin tests was achieved in many cases. The CCD blocker did not affect test results where CCDs were not involved.

Conclusion

Eliminating the effect of IgEs directed against CCDs by inhibition leads to a significant reduction in false-positive in vitro test results without lowering sensitivity towards relevant sensitizations. Application of the CCD blocker may be worthwhile wherever natural allergen extracts or components are used.

Metabarcoding for the parallel identification of several hundred predators and their prey: Application to bat species diet analysis

Assessing diet variability is of main importance to better understand the biology of bats and design conservation strategies. Although the advent of metabarcoding has facilitated such analyses, this approach does not come without challenges. Biases may occur throughout the whole experiment, from fieldwork to biostatistics, resulting in the detection of false negatives, false positives or low taxonomic resolution. We detail a rigorous metabarcoding approach based on a short COI minibarcode and two-step PCR protocol enabling the "all at once" taxonomic identification of bats and their arthropod prey for several hundreds of samples. Our study includes faecal pellets collected in France from 357 bats representing 16 species, as well as insect mock communities that mimic bat meals of known composition, negative and positive controls. All samples were analysed using three replicates. We compare the efficiency of DNA extraction methods, and we evaluate the effectiveness of our protocol using identification success, taxonomic resolution, sensitivity and amplification biases. Our parallel identification strategy of predators and prey reduces the risk of mis-assigning prey to wrong predators and decreases the number of molecular steps. Controls and replicates enable to filter the data and limit the risk of false positives, hence guaranteeing high confidence results for both prey occurrence and bat species identification. We validate 551 COI variants from arthropod including 18 orders, 117 family, 282 genus and 290 species. Our method therefore provides a rapid, resolutive and cost-effective screening tool for addressing evolutionary ecological issues or developing "chirosurveillance" and conservation strategies.

False Positives and Other Statistical Errors in Standard Analyses of Eye Movements in Reading

In research on eye movements in reading, it is common to analyze a number of canonical dependent measures to study how the effects of a manipulation unfold over time. Although this gives rise to the well-known multiple comparisons problem, i.e. an inflated probability that the null hypothesis is incorrectly rejected (Type I error), it is accepted standard practice not to apply any correction procedures. Instead, there appears to be a widespread belief that corrections are not necessary because the increase in false positives is too small to matter. To our knowledge, no formal argument has ever been presented to justify this assumption. Here, we report a computational investigation of this issue using Monte Carlo simulations. Our results show that, contrary to conventional wisdom, false positives are increased to unacceptable levels when no corrections are applied. Our simulations also show that counter-measures like the Bonferroni correction keep false positives in check while reducing statistical power only moderately. Hence, there is little reason why such corrections should not be made a standard requirement. Further, we discuss three statistical illusions that can arise when statistical power is low, and we show how power can be improved to prevent these illusions. In sum, our work renders a detailed picture of the various types of statistical errors than can occur in studies of reading behavior and we provide concrete guidance about how these errors can be avoided.

Lessons From Pinocchio: Cues to Deception May Be Highly Exaggerated

Deception researchers widely acknowledge that cues to deception-observable behaviors that may differ between truthful and deceptive messages-tend to be weak. Nevertheless, several deception cues have been reported with unusually large effect sizes, and some researchers have advocated the use of such cues as tools for detecting deceit and assessing credibility in practical contexts. By examining data from empirical deception-cue research and using a series of Monte Carlo simulations, I demonstrate that many estimated effect sizes of deception cues may be greatly inflated by publication bias, small numbers of estimates, and low power. Indeed, simulations indicate the informational value of the present deception literature is quite low, such that it is not possible to determine whether any given effect is real or a false positive. I warn against the hazards of relying on potentially illusory cues to deception and offer some recommendations for improving the state of the science of deception.

High-Performance Deep Neural Network-Based Tomato Plant Diseases and Pests Diagnosis System With Refinement Filter Bank

A fundamental problem that confronts deep neural networks is the requirement of a large amount of data for a system to be efficient in complex applications. Promising results of this problem are made possible through the use of techniques such as data augmentation or transfer learning of pre-trained models in large datasets. But the problem still persists when the application provides limited or unbalanced data. In addition, the number of false positives resulting from training a deep model significantly cause a negative impact on the performance of the system. This study aims to address the problem of false positives and class unbalance by implementing a Refinement Filter Bank framework for Tomato Plant Diseases and Pests Recognition. The system consists of three main units: First, a Primary Diagnosis Unit (Bounding Box Generator) generates the bounding boxes that contain the location of the infected area and class. The promising boxes belonging to each class are then used as input to a Secondary Diagnosis Unit (CNN Filter Bank) for verification. In this second unit, misclassified samples are filtered through the training of independent CNN classifiers for each class. The result of the CNN Filter Bank is a decision of whether a target belongs to the category as it was detected (True) or not (False) otherwise. Finally, an integration unit combines the information from the primary and secondary units while keeping the True Positive samples and eliminating the False Positives that were misclassified in the first unit. By this implementation, the proposed approach is able to obtain a recognition rate of approximately 96%, which represents an improvement of 13% compared to our previous work in the complex task of tomato diseases and pest recognition. Furthermore, our system is able to deal with the false positives generated by the bounding box generator, and class unbalances that appear especially on datasets with limited data.

Cluster failure revisited: Impact of first level design and physiological noise on cluster false positive rates

Methodological research rarely generates a broad interest, yet our work on the validity of cluster inference methods for functional magnetic resonance imaging (fMRI) created intense discussion on both the minutia of our approach and its implications for the discipline. In the present work, we take on various critiques of our work and further explore the limitations of our original work. We address issues about the particular event-related designs we used, considering multiple event types and randomization of events between subjects. We consider the lack of validity found with one-sample permutation (sign flipping) tests, investigating a number of approaches to improve the false positive control of this widely used procedure. We found that the combination of a two-sided test and cleaning the data using ICA FIX resulted in nominal false positive rates for all data sets, meaning that data cleaning is not only important for resting state fMRI, but also for task fMRI. Finally, we discuss the implications of our work on the fMRI literature as a whole, estimating that at least 10% of the fMRI studies have used the most problematic cluster inference method (p = .01 cluster defining threshold), and how individual studies can be interpreted in light of our findings. These additional results underscore our original conclusions, on the importance of data sharing and thorough evaluation of statistical methods on realistic null data.

Cost of breast-related care in the year following false positive screening mammograms

OBJECTIVE

We sought to estimate the direct cost, from the perspective of the health insurer or purchaser, of breast-care services in the year following a false positive screening mammogram compared with a true negative examination.

DESIGN

We identified 21,125 women aged 40 to 80 years enrolled in an integrated healthcare delivery system in Washington State, who participated in screening mammography between January 1, 1998 and July 30, 2002. Pathology and cancer registry data were used to identify breast cancer diagnoses in the year following the screening mammogram. A positive examination was defined as a Breast Imaging Reporting and Data System assessment of 0, 4, or 5. Women with a positive screening mammogram but no breast cancer diagnosed within 1 year were classified as false positives. We used diagnostic and procedure codes in automated health plan data to identify services received in the year following the screening mammogram. Medicare reimbursement rates were applied to all services. We used ordinary least-squares linear regression to estimate the difference in costs following a false positive versus true negative screening mammogram.

RESULTS

False positive results occurred in 9.9% of women; most false positives (87.3%) were followed by breast imaging only. The mean cost of breast-care following a false positive mammogram was $527. This was $503 (95% confidence interval, $490-$515) more than the cost of breast-care services for true negative women.

CONCLUSIONS

The direct costs for breast-related procedures following false positive screening mammograms may contribute substantially to US healthcare spending.

Accurate SNV detection in single cells by transposon-based whole-genome amplification of complementary strands

The boom of single-cell sequencing technologies in the past decade has profoundly expanded our understanding of fundamental biology. Today, tens of thousands of cells can be measured by single-cell RNA-seq in one experiment. However, single-cell DNA-sequencing studies have been limited by false positives and cost. Here we report META-CS, a single-cell whole-genome amplification method that takes advantage of the complementary strands of double-stranded DNA to filter out false positives and reduce sequencing cost. META-CS achieved the highest accuracy in terms of detecting single-nucleotide variations, and provided potential solutions for the identification of other genomic variants, such as insertions, deletions, and structural variations in single cells.

Single-nucleotide variants (SNVs), pertinent to aging and disease, occur sporadically in the human genome, hence necessitating single-cell measurements. However, detection of single-cell SNVs suffers from false positives (FPs) due to intracellular single-stranded DNA damage and the process of whole-genome amplification (WGA). Here, we report a single-cell WGA method termed multiplexed end-tagging amplification of complementary strands (META-CS), which eliminates nearly all FPs by virtue of DNA complementarity, and achieved the highest accuracy thus far. We validated META-CS by sequencing kindred cells and human sperm, and applied it to other human tissues. Investigation of mature single human neurons revealed increasing SNVs with age and potentially unrepaired strand-specific oxidative guanine damage. We determined SNV frequencies along the genome in differentiated single human blood cells, and identified cell type-dependent mutational patterns for major types of lymphocytes.

Development of Strategies to Decrease False Positive Results in Newborn Screening

The expansion of national newborn screening (NBS) programmes has provided significant benefits in the diagnosis and early treatment of several rare, heritable conditions, preventing adverse health outcomes for most affected infants. New technological developments have enabled the implementation of testing panel covering over 50 disorders. Consequently, the increment of false positive rate has led to a high number of healthy infants recalled for expensive and often invasive additional testing, opening a debate about the harm-benefit ratio of the expanded newborn screening. The false-positive rate represents a challenge for healthcare providers working in NBS systems. Here, we give an overview on the most commonly used strategies for decreasing the adverse effects due to inconclusive screening results. The focus is on NBS performance improvement through the implementation of analytical methods, the application of new and more informative biomarkers, and by using post-analytical interpretive tools. These strategies, used as part of the NBS process, can to enhance the positive predictive value of the test and reduce the parental anxiety and healthcare costs related to the unnecessary tests and procedures.

Diagnostic criteria as dysfunction indicators: bridging the chasm between the definition of mental disorder and diagnostic criteria for specific disorders

According to the introduction to the Diagnostic and Statistical Manual of Mental Disorders (DSM), Fifth Edition, each disorder must satisfy the definition of mental disorder, which requires the presence of both harm and dysfunction. Constructing criteria sets to require harm is relatively straightforward. However, establishing the presence of dysfunction is necessarily inferential because of the lack of knowledge of internal psychological and biological processes and their functions and dysfunctions. Given that virtually every psychiatric symptom characteristic of a DSM disorder can occur under some circumstances in a normally functioning person, diagnostic criteria based on symptoms must be constructed so that the symptoms indicate an internal dysfunction, and are thus inherently pathosuggestive. In this paper, we review strategies used in DSM criteria sets for increasing the pathosuggestiveness of symptoms to ensure that the disorder meets the requirements of the definition of mental disorder. Strategies include the following: requiring a minimum duration and persistence; requiring that the frequency or intensity of a symptom exceed that seen in normal people; requiring disproportionality of symptoms, given the context; requiring pervasiveness of symptom expression across contexts; adding specific exclusions for contextual scenarios in which symptoms are best understood as normal reactions; combining symptoms to increase cumulative pathosuggestiveness; and requiring enough symptoms from an overall syndrome to meet a minimum threshold of pathosuggestiveness. We propose that future revisions of the DSM consider systematic implementation of these strategies in the construction and revision of criteria sets, with the goal of maximizing the pathosuggestiveness of diagnostic criteria to reduce the potential for diagnostic false positives.

Epidemiology, public health, and the rhetoric of false positives

BACKGROUND

As an observational science, epidemiology is regarded by some researchers as inherently flawed and open to false results. In a recent paper, Boffetta et al. [Boffetta P, McLaughlin JK, LaVecchia C, Tarone RE, Lipworth L, Blot WJ. False-positive results in cancer epidemiology: a plea for epistemological modesty. J Natl Cancer Inst 100:988-995 (2008)] argued that "epidemiology is particularly prone to the generation of false-positive results." They also said "the tendency to emphasize and over-interpret what appear to be new findings is commonplace, perhaps in part because of a belief that the findings provide information that may ultimately improve public health" and that "this tendency to hype new findings increases the likelihood of downplaying inconsistencies within the data or any lack of concordance with other sources of evidence." The authors supported these serious charges against epidemiology and epidemiologists with few examples. Although we acknowledge that false positives do occur, we view the position of Boffetta and colleagues on false positives as unbalanced and potentially harmful to public health.

OBJECTIVE

We aim to provide a more balanced evaluation of epidemiology and its contribution to public health discourse.

DISCUSSION

Boffetta and colleagues ignore the fact that false negatives may arise from the very processes that they tout as generating false-positive results. We further disagree with their proposition that false-positive results from a single study will lead to faulty decision making in matters of public health importance. In practice, such public health evaluations are based on all the data available from all relevant disciplines and never to our knowledge on a single study.

CONCLUSIONS

The lack of balance by Boffetta and colleagues in their evaluation of the impact of false-positive findings on epidemiology, the charge that "methodological vigilance is often absent" in epidemiologists' interpretation of their own results, and the false characterization of how epidemiologic findings are used in societal decision making all undermine a major source of information regarding disease risks. We reaffirm the importance of epidemiologic evidence as a critical component of the foundation of public health protection.

Improving the reliability of eDNA data interpretation

Global declines in biodiversity highlight the need to effectively monitor the density and distribution of threatened species. In recent years, molecular survey methods detecting DNA released by target-species into their environment (eDNA) have been rapidly on the rise. Despite providing new, cost-effective tools for conservation, eDNA-based methods are prone to errors. Best field and laboratory practices can mitigate some, but the risks of errors cannot be eliminated and need to be accounted for. Here, we synthesize recent advances in data processing tools that increase the reliability of interpretations drawn from eDNA data. We review advances in occupancy models to consider spatial data-structures and simultaneously assess rates of false positive and negative results. Further, we introduce process-based models and the integration of metabarcoding data as complementing approaches to increase the reliability of target-species assessments. These tools will be most effective when capitalizing on multi-source data sets collating eDNA with classical survey and citizen-science approaches, paving the way for more robust decision-making processes in conservation planning.

Use of high-throughput mass spectrometry to reduce false positives in protease uHTS screens

As a label-free technology, mass spectrometry (MS) enables assays to be generated that monitor the conversion of substrates with native sequences to products without the requirement for substrate modifications or indirect detection methods. Although traditional liquid chromatography (LC)-MS methods are relatively slow for a high-throughput screening (HTS) paradigm, with cycle times typically ≥ 60 s per sample, the Agilent RapidFire High-Throughput Mass Spectrometry (HTMS) System, with a cycle time of 5-7 s per sample, enables rapid analysis of compound numbers compatible with HTS. By monitoring changes in mass directly, HTMS assays can be used as a triaging tool by eliminating large numbers of false positives resulting from fluorescent compound interference or from compounds interacting with hydrophobic fluorescent dyes appended to substrates. Herein, HTMS assays were developed for multiple protease programs, including cysteine, serine, and aspartyl proteases, and applied as a confirmatory assay. The confirmation rate for each protease assay averaged <30%, independent of the primary assay technology used (i.e., luminescent, fluorescent, and time-resolved fluorescent technologies). Importantly, >99% of compounds designed to inhibit the enzymes were confirmed by the corresponding HTMS assay. Hence, HTMS is an effective tool for removing detection-based false positives from ultrahigh-throughput screening, resulting in hit lists enriched in true actives for downstream dose response titrations and hit-to-lead efforts.