Research wranglers: initiatives to improve reproducibility of study findings

Censoring Trace-Level Environmental Data: Statistical Analysis Considerations to Limit Bias

Trace-level environmental data typically include values near or below detection and quantitation thresholds where health effects may result from low-concentration exposures to one chemical over time or to multiple chemicals. In a cook stove case study, bias in dibenzo[a,h]anthracene concentration means and standard deviations (SDs) was assessed following censoring at thresholds for selected analysis approaches: substituting threshold/2, maximum likelihood estimation, robust regression on order statistics, Kaplan-Meier, and omitting censored observations. Means and SDs for gas chromatography-mass spectrometry-determined concentrations were calculated after censoring at detection and calibration thresholds, 17% and 55% of the data, respectively. Threshold/2 substitution was the least biased. Measurement values were subsequently simulated from two log-normal distributions at two sample sizes. Means and SDs were calculated for 30%, 50%, and 80% censoring levels and compared to known distribution counterparts. Simulation results illustrated (1) threshold/2 substitution to be inferior to modern after-censoring statistical approaches and (2) all after-censoring approaches to be inferior to including all measurement data in analysis. Additionally, differences in stove-specific group means were tested for uncensored samples and after censoring. Group differences of means tests varied depending on censoring and distributional decisions. Investigators should guard against censoring-related bias from (explicit or implicit) distributional and analysis approach decisions.

Strategies for robust and accurate experimental approaches to quantify nanomaterial bioaccumulation across a broad range of organisms

One of the key components for environmental risk assessment of engineered nanomaterials (ENMs) is data on bioaccumulation potential. Accurately measuring bioaccumulation can be critical for regulatory decision making regarding material hazard and risk, and for understanding the mechanism of toxicity. This perspective provides expert guidance for performing ENM bioaccumulation measurements across a broad range of test organisms and species. To accomplish this aim, we critically evaluated ENM bioaccumulation within three categories of organisms: single-celled species, multicellular species excluding plants, and multicellular plants. For aqueous exposures of suspended single-celled and small multicellular species, it is critical to perform a robust procedure to separate suspended ENMs and small organisms to avoid overestimating bioaccumulation. For many multicellular organisms, it is essential to differentiate between the ENMs adsorbed to external surfaces or in the digestive tract and the amount absorbed across epithelial tissues. For multicellular plants, key considerations include how exposure route and the role of the rhizosphere may affect the quantitative measurement of uptake, and that the efficiency of washing procedures to remove loosely attached ENMs to the roots is not well understood. Within each organism category, case studies are provided to illustrate key methodological considerations for conducting robust bioaccumulation experiments for different species within each major group. The full scope of ENM bioaccumulation measurements and interpretations are discussed including conducting the organism exposure, separating organisms from the ENMs in the test media after exposure, analytical methods to quantify ENMs in the tissues or cells, and modeling the ENM bioaccumulation results. One key finding to improve bioaccumulation measurements was the critical need for further analytical method development to identify and quantify ENMs in complex matrices. Overall, the discussion, suggestions, and case studies described herein will help improve the robustness of ENM bioaccumulation studies.

The Harm Done to Reproducibility by the Culture of Null Hypothesis Significance Testing

In the last few years, stakeholders in the scientific community have raised alarms about a perceived lack of reproducibility of scientific results. In reaction, guidelines for journals have been promulgated and grant applicants have been asked to address the rigor and reproducibility of their proposed projects. Neither solution addresses a primary culprit, which is the culture of null hypothesis significance testing that dominates statistical analysis and inference. In an innovative research enterprise, selection of results for further evaluation based on null hypothesis significance testing is doomed to yield a low proportion of reproducible results and a high proportion of effects that are initially overestimated. In addition, the culture of null hypothesis significance testing discourages quantitative adjustments to account for systematic errors and quantitative incorporation of prior information. These strategies would otherwise improve reproducibility and have not been previously proposed in the widely cited literature on this topic. Without discarding the culture of null hypothesis significance testing and implementing these alternative methods for statistical analysis and inference, all other strategies for improving reproducibility will yield marginal gains at best.

Irreproducibility in Preclinical Biomedical Research: Perceptions, Uncertainties, and Knowledge Gaps

Concerns regarding the reliability of biomedical research outcomes were precipitated by two independent reports from the pharmaceutical industry that documented a lack of reproducibility in preclinical research in the areas of oncology, endocrinology, and hematology. Given their potential impact on public health, these concerns have been extensively covered in the media. Assessing the magnitude and scope of irreproducibility is limited by the anecdotal nature of the initial reports and a lack of quantitative data on specific failures to reproduce published research. Nevertheless, remediation activities have focused on needed enhancements in transparency and consistency in the reporting of experimental methodologies and results. While such initiatives can effectively bridge knowledge gaps and facilitate best practices across established and emerging research disciplines and therapeutic areas, concerns remain on how these improve on the historical process of independent replication in validating research findings and their potential to inhibit scientific innovation.

Completeness of Follow-Up Determines Validity of Study Findings: Results of a Prospective Repeated Measures Cohort Study

Background

Current reporting guidelines do not call for standardised declaration of follow-up completeness, although study validity depends on the representativeness of measured outcomes. The Follow-Up Index (FUI) describes follow-up completeness at a given study end date as ratio between the investigated and the potential follow-up period. The association between FUI and the accuracy of survival-estimates was investigated.

Methods

FUI and Kaplan-Meier estimates were calculated twice for 1207 consecutive patients undergoing aortic repair during an 11-year period: in a scenario A the population’s clinical routine follow-up data (available from a prospective registry) was analysed conventionally. For the control scenario B, an independent survey was completed at the predefined study end. To determine the relation between FUI and the accuracy of study findings, discrepancies between scenarios regarding FUI, follow-up duration and cumulative survival-estimates were evaluated using multivariate analyses.

Results

Scenario A noted 89 deaths (7.4%) during a mean considered follow-up of 30±28months. Scenario B, although analysing the same study period, detected 304 deaths (25.2%, P<0.001) as it scrutinized the complete follow-up period (49±32months). FUI (0.57±0.35 versus 1.00±0, P<0.001) and cumulative survival estimates (78.7% versus 50.7%, P<0.001) differed significantly between scenarios, suggesting that incomplete follow-up information led to underestimation of mortality. Degree of follow-up completeness (i.e. FUI-quartiles and FUI-intervals) correlated directly with accuracy of study findings: underestimation of long-term mortality increased almost linearly by 30% with every 0.1 drop in FUI (adjusted HR 1.30; 95%-CI 1.24;1.36, P<0.001).

Conclusion

Follow-up completeness is a pre-requisite for reliable outcome assessment and should be declared systematically. FUI represents a simple measure suited as reporting standard. Evidence lacking such information must be challenged as potentially flawed by selection bias.

Raising the bar for reproducible science at the U.S. Environmental Protection Agency Office of Research and Development

Considerable concern has been raised regarding research reproducibility both within and outside the scientific community. Several factors possibly contribute to a lack of reproducibility, including a failure to adequately employ statistical considerations during study design, bias in sample selection or subject recruitment, errors in developing data inclusion/exclusion criteria, and flawed statistical analysis. To address some of these issues, several publishers have developed checklists that authors must complete. Others have either enhanced statistical expertise on existing editorial boards, or formed distinct statistics editorial boards. Although the U.S. Environmental Protection Agency, Office of Research and Development, already has a strong Quality Assurance Program, an initiative was undertaken to further strengthen statistics consideration and other factors in study design and also to ensure these same factors are evaluated during the review and approval of study protocols. To raise awareness of the importance of statistical issues and provide a forum for robust discussion, a Community of Practice for Statistics was formed in January 2014. In addition, three working groups were established to develop a series of questions or criteria that should be considered when designing or reviewing experimental, observational, or modeling focused research. This article describes the process used to develop these study design guidance documents, their contents, how they are being employed by the Agency’s research enterprise, and expected benefits to Agency science. The process and guidance documents presented here may be of utility for any research enterprise interested in enhancing the reproducibility of its science.