Issues with data and analyses: Errors, underlying themes, and potential solutions

An Overview of Reviews on the Association of Low Calorie Sweetener Consumption With Body Weight and Adiposity

BACKGROUND

Numerous systematic reviews (SR) and meta-analyses (MA) on low calorie sweeteners (LCS) have been published in recent years, concluding that LCS have beneficial, neutral, or detrimental effects on various health outcomes, depending on the review.

OBJECTIVES

The objective of this overview of reviews was to determine how the methodologies of SR investigating the association between LCS consumption and body weight (BW) influence their findings and whether MA results can provide a consistent estimated effect.

METHODS

Systematic searches of PubMed, Scopus, and Cochrane Library were conducted in November 2022 to identify SR of randomized controlled trials (RCT) or non-randomized studies (NRS) investigating the association between LCS consumption and BW. The methods, MA results, and conclusions were extracted from each eligible SR.

RESULTS

Of the 985 search results, 20 SR evaluated the association between LCS and BW, drawing from publications of 75 RCT, 42 prospective cohort studies, and 10 cross-sectional studies. There was a considerable lack of overlap of studies included within each SR attributed, in part, to the inclusion of studies based on design; thus, each SR synthesized results from distinctly different studies. Evidence synthesis methods were heterogeneous and often opaque, making it difficult to determine why results from certain studies were excluded or why disparate results were observed between SR.

CONCLUSIONS

SR investigating the effect of LCS on BW implement different methodologies to answer allegedly the same question, drawing from a different set of heterogeneous studies, ignoring the basic assumptions required for MA, resulting in disparate results and conclusions. Previous MA show the large effects of study design, which results in inconsistent estimates of the effect of LCS on BW between MA of RCT and NRS. Given the availability of long-term RCT, these studies should be the basis of determining causal relationships (or lack thereof) between LCS and BW. This trial was registered at PROSPERO as CRD42022351200.

Verification of nucleotide sequence reagent identities in original publications in high impact factor cancer research journals

Human gene research studies that describe wrongly identified nucleotide sequence reagents have been mostly identified in journals of low to moderate impact factor, where unreliable findings could be considered to have limited influence on future research. This study examined whether papers describing wrongly identified nucleotide sequences are also published in high-impact-factor cancer research journals. We manually verified nucleotide sequence identities in original Molecular Cancer articles published in 2014, 2016, 2018, and 2020, including nucleotide sequence reagents that were claimed to target circRNAs. Using keywords identified in some 2018 and 2020 Molecular Cancer papers, we also verified nucleotide sequence identities in 2020 Oncogene papers that studied miRNA(s) and/or circRNA(s). Overall, 3.8% (251/6647) and 4.0% (47/1165) nucleotide sequences that were verified in Molecular Cancer and Oncogene papers, respectively, were found to be wrongly identified. Wrongly identified nucleotide sequences were distributed across 18% (91/500) original Molecular Cancer papers, including 38% (31/82) Molecular Cancer papers from 2020, and 40% (21/52) selected Oncogene papers from 2020. Original papers with wrongly identified nucleotide sequences were therefore unexpectedly frequent in two high-impact-factor cancer research journals, highlighting the risks of employing journal impact factors or citations as proxies for research quality.

Identification of application and interpretation errors that can occur in pairwise meta-analyses in systematic reviews of interventions: a systematic review

OBJECTIVES

To generate a bank of items describing application and interpretation errors that can arise in pairwise meta-analyses in systematic reviews of interventions.

STUDY DESIGN AND SETTING

MEDLINE, Embase, and Scopus were searched to identify studies describing types of errors in meta-analyses. Descriptions of errors and supporting quotes were extracted by multiple authors. Errors were reviewed at team meetings to determine if they should be excluded, reworded, or combined with other errors, and were categorized into broad categories of errors and subcategories within.

RESULTS

Fifty articles met our inclusion criteria, leading to the identification of 139 errors. We identified 25 errors covering data extraction/manipulation, 74 covering statistical analyses, and 40 covering interpretation. Many of the statistical analysis errors related to the meta-analysis model (eg, using a two-stage strategy to determine whether to select a fixed or random-effects model) and statistical heterogeneity (eg, not undertaking an assessment for statistical heterogeneity).

CONCLUSION

We generated a comprehensive bank of possible errors that can arise in the application and interpretation of meta-analyses in systematic reviews of interventions. This item bank of errors provides the foundation for developing a checklist to help peer reviewers detect statistical errors.

Advancing data honesty in experimental biology

The ease with which scientific data, particularly certain types of raw data in experimental biology, can be fabricated without trace begs urgent attention. This is thought to be a widespread problem across the academic world, where published results are the major currency, incentivizing publication of (usually positive) results at the cost of lax scientific rigor and even fraudulent data. Although solutions to improve data sharing and methodological transparency are increasingly being implemented, the inability to detect dishonesty within raw data remains an inherent flaw in the way in which we judge research. We therefore propose that one solution would be the development of a non-modifiable raw data format that could be published alongside scientific results; a format that would enable data authentication from the earliest stages of experimental data collection. A further extension of this tool could allow changes to the initial original version to be tracked, so every reviewer and reader could follow the logical footsteps of the author and detect unintentional errors or intentional manipulations of the data. Were such a tool to be developed, we would not advocate its use as a prerequisite for journal submission; rather, we envisage that authors would be given the option to provide such authentication. Only authors who did not manipulate or fabricate their data can provide the original data without risking discovery, so the mere choice to do so already increases their credibility (much like 'honest signaling' in animals). We strongly believe that such a tool would enhance data honesty and encourage more reliable science.

On hidden factors and design-associated errors that may lead to data misinterpretation: An example from preclinical research on the potential seasonality of neonatal seizures

Unintentional misinterpretation of research in published biomedical reports that is not based on statistical flaws is often underrecognized, despite its possible impact on science, clinical practice, and public health. Important causes of such misinterpretation of scientific data, resulting in either false positive or false negative conclusions, include design-associated errors and hidden (or latent) variables that are not easily recognized during data analysis. Furthermore, cognitive biases, such as the inclination to seek patterns in data whether they exist or not, may lead to misinterpretation of data. Here, we give an example of these problems from hypothesis-driven research on the potential seasonality of neonatal seizures in a rat model of birth asphyxia. This commentary aims to raise awareness among the general scientific audience about the issues related to the presence of unintentional misinterpretation in published reports.

Originating Research Ideas

Research aimed at generating new knowledge is the heart of the scholarship of discovery. The author of this paper explores how original research ideas can be generated for formal investigations and artsciencing. Curiosity and creativity are presented as "seeds" for originating ideas, and seven patterns (adjacent possible, liquid networks, the slow hunch, serendipity, error, exaptation, and platforms) are described as synergistic potentiators for geminating original research ideas.

'One size doesn't fit all': Lessons from interaction analysis on tailoring Open Science practices to qualitative research

The Open Science Movement aims to enhance the soundness, transparency, and accessibility of scientific research, and at the same time increase public trust in science. Currently, Open Science practices are mainly presented as solutions to the 'reproducibility crisis' in hypothetico-deductive quantitative research. Increasing interest has been shown towards exploring how these practices can be adopted by qualitative researchers. In reviewing this emerging body of work, we conclude that the issue of diversity within qualitative research has not been adequately addressed. Furthermore, we find that many of these endeavours start with existing solutions for which they are trying to find matching problems to be solved. We contrast this approach with a natural incorporation of Open Science practices within interaction analysis and its constituent research traditions: conversation analysis, discursive psychology, ethnomethodology, and membership categorisation analysis. Zooming in on the development of conversation analysis starting in the 1960s, we highlight how practices for opening up and sharing data and analytic thinking have been embedded into its methodology. On the basis of this presentation, we propose a series of lessons learned for adopting Open Science practices in qualitative research.

Facilitators for and Barriers to the Implementation of Performance Accountability Mechanisms for Quality Improvement in the Delivery of Maternal Health Services in a District Hospital in Pwani Region, Tanzania

Tanzania experiences a burden of maternal mortality and morbidity. Despite the efforts to institute accountability mechanisms, little is known about quality improvement in the delivery of maternal health services. This study aimed at exploring barriers and facilitators to enforcing performance accountability mechanisms for quality improvement in maternal health services. A case study design was used to conduct semi-structured interviews with thirteen key informants. Data were analyzed using thematic analyses. The findings were linked to two main performance accountability mechanisms: maternal and perinatal death reviews (MPDRs) and monitoring and evaluation (M&E). Prioritization of the maternal health agenda by the government and the presence of maternal death review committees were the main facilitators for MPDRs, while negligence, inadequate follow-up, poor record-keeping, and delays were the main barriers facing MPDRs. M&E was facilitated by the availability of health management information systems, day-to-day ward rounds, online ordering of medicines, and the use of biometrics. Non-use of data for decision-making, supervision being performed on an ad hoc basis, and inadequate health workforce were the main barriers to M&E. The findings underscore that barriers to the performance accountability mechanisms are systemic and account for limited effectiveness in the improvement of quality of care.

BatMan: Mitigating Batch Effects Via Stratification for Survival Outcome Prediction

Reproducible translation of transcriptomics data has been hampered by the ubiquitous presence of batch effects. Statistical methods for managing batch effects were initially developed in the setting of sample group comparison and later borrowed for other settings such as survival outcome prediction. The most notable such method is ComBat, which adjusts for batches by including it as a covariate alongside sample groups in a linear regression. In survival prediction, however, ComBat is used without definable groups for survival outcome and is done sequentially with survival regression for a potentially batch-confounded outcome. To address these issues, we propose a new method called BATch MitigAtion via stratificatioN (BatMan). It adjusts batches as strata in survival regression and uses variable selection methods such as the regularized regression to handle high dimensionality. We assess the performance of BatMan in comparison with ComBat, each used either alone or in conjunction with data normalization, in a resampling-based simulation study under various levels of predictive signal strength and patterns of batch-outcome association. Our simulations show that (1) BatMan outperforms ComBat in nearly all scenarios when there are batch effects in the data and (2) their performance can be worsened by the addition of data normalization. We further evaluate them using microRNA data for ovarian cancer from the Cancer Genome Atlas and find that BatMan outforms ComBat while the addition of data normalization worsens the prediction. Our study thus shows the advantage of BatMan and raises caution about the use of data normalization in the context of developing survival prediction models. The BatMan method and the simulation tool for performance assessment are implemented in R and publicly available at LXQin/PRECISION.survival-GitHub.

Teaching students to R3eason, not merely to solve problem sets: The role of philosophy and visual data communication in accessible data science education

Much guidance on statistical training in STEM fields has been focused largely on the undergraduate cohort, with graduate education often being absent from the equation. Training in quantitative methods and reasoning is critical for graduate students in biomedical and science programs to foster reproducible and responsible research practices. We argue that graduate student education should more center around fundamental reasoning and integration skills rather than mainly on listing 1 statistical test method after the other without conveying the bigger context picture or critical argumentation skills that will enable student to improve research integrity through rigorous practice. Herein, we describe the approach we take in a quantitative reasoning course in the R3 program at the Johns Hopkins Bloomberg School of Public Health, with an error-focused lens, based on visualization and communication competencies. Specifically, we take this perspective stemming from the discussed causes of irreproducibility and apply it specifically to the many aspects of good statistical practice in science, ranging from experimental design to data collection and analysis, and conclusions drawn from the data. We also provide tips and guidelines for the implementation and adaptation of our course material to various graduate biomedical and STEM science programs.

Social determinants of health derived from people with opioid use disorder: Improving data collection, integration and use with cross-domain collaboration and reproducible, data-centric, notebook-style workflows

Deriving social determinants of health from underserved populations is an important step in the process of improving the well-being of these populations and in driving policy improvements to facilitate positive change in health outcomes. Collection, integration, and effective use of clinical data for this purpose presents a variety of specific challenges. We assert that combining expertise from three distinct domains, specifically, medical, statistical, and computer and data science can be applied along with provenance-aware, self-documenting workflow tools. This combination permits data integration and facilitates the creation of reproducible workflows and usable (reproducible) results from the sensitive and disparate sources of clinical data that exist for underserved populations.

Market Analysis with Business Intelligence System for Marketing Planning

The automotive and auto parts industries are important economic sectors in Thailand. With rapidly changing technology, every organization should understand what needs to be improved clearly, and shift their strategies to meet evolving consumer demands. The purpose of this research is to develop a Business Intelligence system for a brake pad manufacturing company in Thailand. By analyzing the relationship between market demand and supply components of the company through regression analysis and the principles of the marketing mix, we develop a product lifecycle curve for forecasting product sales. The developed system increases the workflow efficiency of the case study company, being able to simplify the traditional data preparation process that requires employees to collect and summarize data every time a request is made. An intelligence dashboard is subsequently created to help support decision-making, facilitate communication within the company, and eventually improve team efficiency and productivity.

Thirty-three myths and misconceptions about population data: from data capture and processing to linkage

Databases covering all individuals of a population are increasingly used for research and decision-making. The massive size of such databases is often mistaken as a guarantee for valid inferences. However, population data have characteristics that make them challenging to use. Various assumptions on population coverage and data quality are commonly made, including how such data were captured and what types of processing have been applied to them. Furthermore, the full potential of population data can often only be unlocked when such data are linked to other databases. Record linkage often implies subtle technical problems, which are easily missed. We discuss a diverse range of myths and misconceptions relevant for anybody capturing, processing, linking, or analysing population data. Remarkably, many of these myths and misconceptions are due to the social nature of data collections and are therefore missed by purely technical accounts of data processing. Many are also not well documented in scientific publications. We conclude with a set of recommendations for using population data.

Synaptic degeneration in Alzheimer disease

Alzheimer disease (AD) is characterized by progressive cognitive decline in older individuals accompanied by the presence of two pathological protein aggregates - amyloid-β and phosphorylated tau - in the brain. The disease results in brain atrophy caused by neuronal loss and synapse degeneration. Synaptic loss strongly correlates with cognitive decline in both humans and animal models of AD. Indeed, evidence suggests that soluble forms of amyloid-β and tau can cause synaptotoxicity and spread through neural circuits. These pathological changes are accompanied by an altered phenotype in the glial cells of the brain - one hypothesis is that glia excessively ingest synapses and modulate the trans-synaptic spread of pathology. To date, effective therapies for the treatment or prevention of AD are lacking, but understanding how synaptic degeneration occurs will be essential for the development of new interventions. Here, we highlight the mechanisms through which synapses degenerate in the AD brain, and discuss key questions that still need to be answered. We also cover the ways in which our understanding of the mechanisms of synaptic degeneration is leading to new therapeutic approaches for AD.

Protection of the human gene research literature from contract cheating organizations known as research paper mills

Human gene research generates new biology insights with translational potential, yet few studies have considered the health of the human gene literature. The accessibility of human genes for targeted research, combined with unreasonable publication pressures and recent developments in scholarly publishing, may have created a market for low-quality or fraudulent human gene research articles, including articles produced by contract cheating organizations known as paper mills. This review summarises the evidence that paper mills contribute to the human gene research literature at scale and outlines why targeted gene research may be particularly vulnerable to systematic research fraud. To raise awareness of targeted gene research from paper mills, we highlight features of problematic manuscripts and publications that can be detected by gene researchers and/or journal staff. As improved awareness and detection could drive the further evolution of paper mill-supported publications, we also propose changes to academic publishing to more effectively deter and correct problematic publications at scale. In summary, the threat of paper mill-supported gene research highlights the need for all researchers to approach the literature with a more critical mindset, and demand publications that are underpinned by plausible research justifications, rigorous experiments and fully transparent reporting.

Peer review and the pillar of salt: a case study

Peer review has long been regarded as the gold standard of scientific publication, essential to the integrity of science itself. But, as any publishing scientist knows, peer review has its downside, including long delays and reviewer bias. Until the coming of the Internet, there appeared to be no alternative. Now, articles appear online as preprints almost immediately upon submission. But they lack peer review and thus their scientific standing can be questioned. Post-publication discussion platforms such as PubPeer have proven useful, but are no substitute for pre-publication peer review. Nevertheless, some may be tempted to believe that peer review can now be done without. This article challenges that view by analyzing a recent, non-peer-reviewed article in Skeptical Enquirer, a magazine published by the Committee for Skeptical Enquiry (CSI). The article, “Sodom Meteor Strike Claims Should Be Taken With a Pillar of Salt,” casts doubt on one of the most widely read scientific articles of the last decade and provides a stern warning of the cost of abandoning peer review.

Practical notes on popular statistical tests in renal physiology

Competent statistical analysis is essential to maintain rigor and reproducibility in physiological research. Unfortunately, the benefits offered by statistics are often negated by misuse or inadequate reporting of statistical methods. To address the need for improved quality of statistical analysis in papers, the American Physiological Society released guidelines for reporting statistics in journals published by the society. The guidelines reinforce high standards for the presentation of statistical data in physiology but focus on the conceptual challenges and, thus, may be of limited use to an unprepared reader. Experimental scientists working in the renal field may benefit from putting the existing guidelines in a practical context. This paper discusses the application of widespread hypothesis tests in a confirmatory study. We simulated pharmacological experiments assessing intracellular calcium in cultured renal cells and kidney function at the systemic level to review best practices for data analysis, graphical presentation, and reporting. Such experiments are ubiquitously used in renal physiology and could be easily translated to other practical applications to fit the reader's specific needs. We provide step-by-step guidelines for using the most common types of t tests and ANOVA and discuss typical mistakes associated with them. We also briefly consider normality tests, exclusion criteria, and identification of technical and experimental replicates. This review is supposed to help the reader analyze, illustrate, and report the findings correctly and will hopefully serve as a gauge for a level of design complexity when it might be time to consult a biostatistician.

Data transforming: A concept analysis

PURPOSE

The aims of this study are to clarify the concept of how data retrieved from electronic health records (EHR) are transformed into nurses' tacit knowledge for evidence-based practice from a cognitive perspective at a macro-organizational level, and to identify this concept's attributes, antecedents, and consequences in the nursing field.

SOURCE

A literature review was conducted by performing a search on scientific databases using the key terms "data," "transform," "EHR," "nursing," "tacit knowledge," "organization," "data," "interpretation," and "healthcare." Forty-nine articles and four books were selected for the analysis. The process was audited by two independent experts to ensure neutrality and credibility.

CONCLUSION

Data transforming is a complex cognitive process among different groups of data stakeholders at a macro-organizational level. The concept of data transforming has three attributes: analytical, respectful, and social. The antecedents of these attributes are skillful, immersive, and mission-driven. They have either positive or negative consequences for frontline nurses. These findings not only add to the body of knowledge but also serve as an important impetus for further theory development and research in nursing.

Valuing the Diversity of Research Methods to Advance Nutrition Science

The ASN Board of Directors appointed the Nutrition Research Task Force to develop a report on scientific methods used in nutrition science to advance discovery, interpretation, and application of knowledge in the field. The genesis of this report was growing concern about the tone of discourse among nutrition professionals and the implications of acrimony on the productive study and translation of nutrition science. Too often, honest differences of opinion are cast as conflicts instead of areas of needed collaboration. Recognition of the value (and limitations) of contributions from well-executed nutrition science derived from the various approaches used in the discipline, as well as appreciation of how their layering will yield the strongest evidence base, will provide a basis for greater productivity and impact. Greater collaborative efforts within the field of nutrition science will require an understanding that each method or approach has a place and function that should be valued and used together to create the nutrition evidence base. Precision nutrition was identified as an important emerging nutrition topic by the preponderance of task force members, and this theme was adopted for the report because it lent itself to integration of many approaches in nutrition science. Although the primary audience for this report is nutrition researchers and other nutrition professionals, a secondary aim is to develop a document useful for the various audiences that translate nutrition research, including journalists, clinicians, and policymakers. The intent is to promote accurate, transparent, verifiable evidence-based communication about nutrition science. This will facilitate reasoned interpretation and application of emerging findings and, thereby, improve understanding and trust in nutrition science and appropriate characterization, development, and adoption of recommendations.

Achieving greater policy coherence and harmonisation for marine litter management in the North-East Atlantic and Wider Caribbean Region

The effects of marine litter pose ecological, economic and optional value threats to the countries and territories on the margins of the Atlantic Basin. As the abundance of transboundary marine litter increases, this in turn has triggered the development of marine litter policy action plans and inter-regional collaboration. The OSPAR Commission and Cartagena Convention of North-East Atlantic (NEA) and the Wider Caribbean Region (WCR) respectively, have established a memorandum of understanding to facilitate collaboration for enhanced marine litter management. In light of this development, this paper conducts a scoping review of management measures of marine litter within these regions as well as a coherence analysis between their regional action plans (RAPs). The analysis uses a proactive framework based on a modified categorization by Chen (2015) and Williams and Rangel-Buitrago (2019) for comparatively analyse management measures. The analytical categorizations include legislation and enforcement measures, prevention, removal, monitoring, research, information management and exchange and education and outreach. Although specific approaches differed among the regions depending upon geographical and individual situations, each categorization included detailed management activities except for prevention in NEA and removal in the WCR. The comparative analysis demonstrates that there are opportunities for inter-regional cooperation, knowledge sharing and overall enhanced and informed marine litter management.

Paediatric surgical trials, their fragility index, and why to avoid using it to evaluate results

BACKGROUND

The fragility index has been gaining ground in the evaluation of comparative clinical studies. Many scientists evaluated trials in their fields and deemed them to be fragile, although there is no consensus on the definition of fragility. We aimed to calculate the fragility index and its permutations for paediatric surgical trials.

METHODS

We searched pubmed for prospectively conducted paediatric surgical trials with intervention and control group without limitations and calculated their (reverse) fragility indices and respective quotients along with posthoc-power. Relationships between variables were evaluated using Spearman's ρ. We also calculated S values by negative log transformation base-2 of P values.

RESULTS

Of 516 retrieved records, we included 87. The median fragility index was 1.5 (interquartile range: 0-4) and the median reverse fragility index was 3 (interquartile range: 2-4), although they were statistically not different (Mood's test: χ² = 0.557, df = 1, P = 0.4556). P values and fragility indices were strongly inversely correlated (ρ = - 0.71, 95% confidence interval: - 0.53 to - 0.85, P < 0.0001), while reverse fragility indices were moderately correlated to P values (ρ = 0.5, 95% confidence interval: 0.37-0.62, P < 0.0001). A fragility index of 1 resulted from P values between 0.039 and 0.003, which resulted in S values between 4 and 8.

CONCLUSIONS

Fragility indices, reverse fragility indices, and their respective fragility quotients of paediatric surgical trials are low. The fragility index can be viewed as no more than a transformed P value with even more substantial limitations. Its inherent penalisation of small studies irrespective of their clinical relevance is particularly harmful for paediatric surgery. Consequently, the fragility index should be avoided.

Reproducibility in the Social Sciences

Concern over social scientists' inability to reproduce empirical research has spawned a vast and rapidly growing literature. The size and growth of this literature make it difficult for newly interested academics to come up to speed. Here, we provide a formal text modeling approach to characterize the entirety of the field, which allows us to summarize the breadth of this literature and identify core themes. We construct and analyze text networks built from 1,947 articles to reveal differences across social science disciplines within the body of reproducibility publications and to discuss the diversity of subtopics addressed in the literature. This field-wide view suggests that reproducibility is a heterogeneous problem with multiple sources for errors and strategies for solutions, a finding that is somewhat at odds with calls for largely passive remedies reliant on open science. We propose an alternative rigor and reproducibility model that takes an active approach to rigor prior to publication, which may overcome some of the shortfalls of the postpublication model.

How to write statistical analysis section in medical research

Reporting of statistical analysis is essential in any clinical and translational research study. However, medical research studies sometimes report statistical analysis that is either inappropriate or insufficient to attest to the accuracy and validity of findings and conclusions. Published works involving inaccurate statistical analyses and insufficient reporting influence the conduct of future scientific studies, including meta-analyses and medical decisions. Although the biostatistical practice has been improved over the years due to the involvement of statistical reviewers and collaborators in research studies, there remain areas of improvement for transparent reporting of the statistical analysis section in a study. Evidence-based biostatistics practice throughout the research is useful for generating reliable data and translating meaningful data to meaningful interpretation and decisions in medical research. Most existing research reporting guidelines do not provide guidance for reporting methods in the statistical analysis section that helps in evaluating the quality of findings and data interpretation. In this report, we highlight the global and critical steps to be reported in the statistical analysis of grants and research articles. We provide clarity and the importance of understanding study objective types, data generation process, effect size use, evidence-based biostatistical methods use, and development of statistical models through several thematic frameworks. We also provide published examples of adherence or non-adherence to methodological standards related to each step in the statistical analysis and their implications. We believe the suggestions provided in this report can have far-reaching implications for education and strengthening the quality of statistical reporting and biostatistical practice in medical research.

A Synthesis of the Formats for Correcting Erroneous and Fraudulent Academic Literature, and Associated Challenges

Academic publishing is undergoing a highly transformative process, and many established rules and value systems that are in place, such as traditional peer review (TPR) and preprints, are facing unprecedented challenges, including as a result of post-publication peer review. The integrity and validity of the academic literature continue to rely naively on blind trust, while TPR and preprints continue to fail to effectively screen out errors, fraud, and misconduct. Imperfect TPR invariably results in imperfect papers that have passed through varying levels of rigor of screening and validation. If errors or misconduct were not detected during TPR's editorial screening, but are detected at the post-publication stage, an opportunity is created to correct the academic record. Currently, the most common forms of correcting the academic literature are errata, corrigenda, expressions of concern, and retractions or withdrawals. Some additional measures to correct the literature have emerged, including manuscript versioning, amendments, partial retractions and retract and replace. Preprints can also be corrected if their version is updated. This paper discusses the risks, benefits and limitations of these forms of correcting the academic literature.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10838-022-09607-4.

Hiding in Plain Sight: The Issue of Hidden Variables

Here we discuss "hidden variables", which are typically introduced during an experiment as a consequence of the application of two independent variables together to create a stimulus. With increased sophistication in modern chemical biology tools and related precision interrogation techniques, hidden variables have become integral to many chemical biologists' routine experiments. For instance, they can appear in the use of light-activatable chemical probes (e.g., μMap, T-REX), or stimulus-induced enzyme activation (e.g., APEX). Unfortunately, control experiments assess only how independent variables affect measured outcomes and not the multiple differences between the two independent variables and the twain. We outline ways to account for potential hidden variables in experimental design and data interpretation as a means to aid developers of new methods, particularly those involving light-driven techniques, chemical activation, or biorthogonal chemistries, to better incorporate well-controlled procedures.

A System to Easily Manage Metadata in Biomedical Research Labs Based on Open-source Software

In most biomedical labs, researchers gather metadata (i.e., all details about the experimental data) in paper notebooks, spreadsheets, or, sometimes, electronic notebooks. When data analyses occur, the related details usually go into other notebooks or spreadsheets, and more metadata are available. The whole thing rapidly becomes very complex and disjointed, and keeping track of all these things can be daunting. Organizing all the relevant data and related metadata for analysis, publication, sharing, or deposit into archives can be time-consuming, difficult, and prone to errors. By having metadata in a centralized system that contains all details from the start, the process is greatly simplified. While lab management software is available, it can be costly and inflexible. The system described here is based on a popular, freely available, and open-source wiki platform. It provides a simple but powerful way for biomedical research labs to set up a metadata management system linking the whole research process. The system enhances efficiency, transparency, reliability, and rigor, which are key factors to improving reproducibility. The flexibility afforded by the system simplifies implementation of specialized lab requirements and future needs. The protocol presented here describes how to create the system from scratch, how to use it for gathering basic metadata, and provides a fully functional version for perusal by the reader. Graphical abstract: Lab Metadata Management System.

Identification of human gene research articles with wrongly identified nucleotide sequences

Nucleotide sequence reagents underpin molecular techniques that have been applied across hundreds of thousands of publications. We have previously reported wrongly identified nucleotide sequence reagents in human research publications and described a semi-automated screening tool Seek & Blastn to fact-check their claimed status. We applied Seek & Blastn to screen >11,700 publications across five literature corpora, including all original publications in Gene from 2007 to 2018 and all original open-access publications in Oncology Reports from 2014 to 2018. After manually checking Seek & Blastn outputs for >3,400 human research articles, we identified 712 articles across 78 journals that described at least one wrongly identified nucleotide sequence. Verifying the claimed identities of >13,700 sequences highlighted 1,535 wrongly identified sequences, most of which were claimed targeting reagents for the analysis of 365 human protein-coding genes and 120 non-coding RNAs. The 712 problematic articles have received >17,000 citations, including citations by human clinical trials. Given our estimate that approximately one-quarter of problematic articles may misinform the future development of human therapies, urgent measures are required to address unreliable gene research articles.

An observational analysis of the trope "A p-value of < 0.05 was considered statistically significant" and other cut-and-paste statistical methods

Appropriate descriptions of statistical methods are essential for evaluating research quality and reproducibility. Despite continued efforts to improve reporting in publications, inadequate descriptions of statistical methods persist. At times, reading statistical methods sections can conjure feelings of dèjá vu, with content resembling cut-and-pasted or "boilerplate text" from already published work. Instances of boilerplate text suggest a mechanistic approach to statistical analysis, where the same default methods are being used and described using standardized text. To investigate the extent of this practice, we analyzed text extracted from published statistical methods sections from PLOS ONE and the Australian and New Zealand Clinical Trials Registry (ANZCTR). Topic modeling was applied to analyze data from 111,731 papers published in PLOS ONE and 9,523 studies registered with the ANZCTR. PLOS ONE topics emphasized definitions of statistical significance, software and descriptive statistics. One in three PLOS ONE papers contained at least 1 sentence that was a direct copy from another paper. 12,675 papers (11%) closely matched to the sentence "a p-value < 0.05 was considered statistically significant". Common topics across ANZCTR studies differentiated between study designs and analysis methods, with matching text found in approximately 3% of sections. Our findings quantify a serious problem affecting the reporting of statistical methods and shed light on perceptions about the communication of statistics as part of the scientific process. Results further emphasize the importance of rigorous statistical review to ensure that adequate descriptions of methods are prioritized over relatively minor details such as p-values and software when reporting research outcomes.

Between two stools: preclinical research, reproducibility, and statistical design of experiments

Translation of animal-based preclinical research is hampered by poor validity and reproducibility issues. Unfortunately, preclinical research has 'fallen between the stools' of competing study design traditions. Preclinical studies are often characterised by small sample sizes, large variability, and 'problem' data. Although Fisher-type designs with randomisation and blocking are appropriate and have been vigorously promoted, structured statistically-based designs are almost unknown. Traditional analysis methods are commonly misapplied, and basic terminology and principles of inference testing misinterpreted. Problems are compounded by the lack of adequate statistical training for researchers, and failure of statistical educators to account for the unique demands of preclinical research. The solution is a return to the basics: statistical education tailored to non-statistician investigators, with clear communication of statistical concepts, and curricula that address design and data issues specific to preclinical research. Statistics curricula should focus on statistics as process: data sampling and study design before analysis and inference. Properly-designed and analysed experiments are a matter of ethics as much as procedure. Shifting the focus of statistical education from rote hypothesis testing to sound methodology will reduce the numbers of animals wasted in noninformative experiments and increase overall scientific quality and value of published research.

Does Neuromodulation Reduce Chronic Pain Patient Emergency Department Utilization?

BACKGROUND

Chronic pain (CP) affects roughly 100 million adults in the United States. These subjects present disproportionately to the emergency department (ED). Neuromodulation (NM) has been shown to reduce ED visits longitudinally in subjects.

OBJECTIVE

To compare ED utilization rates between subjects with CP with and without NM.

METHODS

Subjects with failed back surgery syndrome, complex regional pain syndrome, or neuropathic pain diagnosis who visited the hospital between January 1, 2019, and December 31, 2019, were included. Subjects were divided into a NM-treated cohort and a non-NM cohort. Demographic information, medications, and pain provider visits were obtained. Pain-related ED visits between 2017 and 2019 were compared.

RESULTS

A total of 2516 subjects were identified; 291 (11.6%) previously underwent NM. The non-NM cohort had significantly higher rate of pain-related ED visits compared with the NM cohort (15.1% vs 10.0%, P = .018). Younger age (odds ratio [OR] = 0.888 [0.843-0.935]), shorter distance to the hospital (OR = 0.807 [0.767-0.849]), lower household income (OR = 0.865 [0.831-0.901]), opioid use (OR = 1.375 [1.291-1.465]), nonopioid use (OR = 1.079 [1.033-1.128]), and non-NM therapy (OR = 1.751 [1.283-2.390]) were significant predictors of ED visits. Opioid use was the only significant predictor (OR = 6.124 [1.417-26.473]) associated with ED visits in the NM cohort.

CONCLUSION

Subjects who underwent NM had fewer visits to the ED when compared with similar subjects who received conventional treatment. Opioid use prompted increased ED utilization in both cohorts. We posit that NM leads to improvement in pain outcomes, integration with multidisciplinary pain specialists, and reduction in severity and frequency of acute pain exacerbations, thereby limiting health care resource utilization.

From Model Organisms to Humans, the Opportunity for More Rigor in Methodologic and Statistical Analysis, Design, and Interpretation of Aging and Senescence Research

This review identifies frequent design and analysis errors in aging and senescence research and discusses best practices in study design, statistical methods, analyses, and interpretation. Recommendations are offered for how to avoid these problems. The following issues are addressed: (a) errors in randomization, (b) errors related to testing within-group instead of between-group differences, (c) failing to account for clustering, (d) failing to consider interference effects, (e) standardizing metrics of effect size, (f) maximum life-span testing, (g) testing for effects beyond the mean, (h) tests for power and sample size, (i) compression of morbidity versus survival curve squaring, and (j) other hot topics, including modeling high-dimensional data and complex relationships and assessing model assumptions and biases. We hope that bringing increased awareness of these topics to the scientific community will emphasize the importance of employing sound statistical practices in all aspects of aging and senescence research.

Evaluating the accuracy and biological meaning of visits to RFID-enabled bird feeders using video

Radio-frequency identification (RFID) technology has gained popularity in ornithological studies as a way to collect large quantities of data to answer specific biological questions, but few published studies report methodologies used for validating the accuracy of RFID data. Further, connections between the RFID data and the behaviors of interest in a study are not always clearly established. These methodological deficiencies may seriously impact a study's results and subsequent interpretation. We built RFID-equipped bird feeders and mounted them at three sites in Tompkins County, New York. We deployed passive integrated transponder tags on black-capped chickadees, tufted titmice, and white-breasted nuthatches and used a GoPro video camera to record the three tagged species at the feeders. We then reviewed the video to determine the accuracy of the RFID reader and understand the birds' behavior at the feeders. We found that our RFID system recorded only 34.2% of all visits by tagged birds (n = 237) and that RFID detection increased with the length of a visit. We also found that our three tagged species and two other species that visited the feeders, American goldfinch and hairy woodpecker, retrieved food in 79.5% of their visits. Chickadees, titmice, nuthatches, and woodpeckers spent, on average, 2.3 s at feeders to collect one seed per visit. In contrast, goldfinches spent an average of 9.0 s at feeders and consumed up to 30 seeds per visit. Our results demonstrate the importance of confirming detection accuracy and that video can be used to identify behavioral characteristics associated with an RFID reader's detections. This simple-yet time-intensive-method for assessing the accuracy and biological meaning of RFID data is useful for ornithological studies but can be used in research focusing on various taxa and study systems.

Errors in the implementation, analysis, and reporting of randomization within obesity and nutrition research: a guide to their avoidance

Randomization is an important tool used to establish causal inferences in studies designed to further our understanding of questions related to obesity and nutrition. To take advantage of the inferences afforded by randomization, scientific standards must be upheld during the planning, execution, analysis, and reporting of such studies. We discuss ten errors in randomized experiments from real-world examples from the literature and outline best practices for their avoidance. These ten errors include: representing nonrandom allocation as random, failing to adequately conceal allocation, not accounting for changing allocation ratios, replacing subjects in nonrandom ways, failing to account for non-independence, drawing inferences by comparing statistical significance from within-group comparisons instead of between-groups, pooling data and breaking the randomized design, failing to account for missing data, failing to report sufficient information to understand study methods, and failing to frame the causal question as testing the randomized assignment per se. We hope that these examples will aid researchers, reviewers, journal editors, and other readers to endeavor to a high standard of scientific rigor in randomized experiments within obesity and nutrition research.

Toward more rigorous and informative nutritional epidemiology: The rational space between dismissal and defense of the status quo

To date, nutritional epidemiology has relied heavily on relatively weak methods including simple observational designs and substandard measurements. Despite low internal validity and other sources of bias, claims of causality are made commonly in this literature. Nutritional epidemiology investigations can be improved through greater scientific rigor and adherence to scientific reporting commensurate with research methods used. Some commentators advocate jettisoning nutritional epidemiology entirely, perhaps believing improvements are impossible. Still others support only normative refinements. But neither abolition nor minor tweaks are appropriate. Nutritional epidemiology, in its present state, offers utility, yet also needs marked, reformational renovation. Changing the status quo will require ongoing, unflinching scrutiny of research questions, practices, and reporting-and a willingness to admit that "good enough" is no longer good enough. As such, a workshop entitled "Toward more rigorous and informative nutritional epidemiology: the rational space between dismissal and defense of the status quo" was held from July 15 to August 14, 2020. This virtual symposium focused on: (1) Stronger Designs, (2) Stronger Measurement, (3) Stronger Analyses, and (4) Stronger Execution and Reporting. Participants from several leading academic institutions explored existing, evolving, and new better practices, tools, and techniques to collaboratively advance specific recommendations for strengthening nutritional epidemiology.

The Tone Debate: Knowledge, Self, and Social Order

In the replication crisis in psychology, a “tone debate” has developed. It concerns the question of how to conduct scientific debate effectively and ethically. How should scientists give critique without unnecessarily damaging relations? The increasing use of Facebook and Twitter by researchers has made this issue especially pressing, as these social technologies have greatly expanded the possibilities for conversation between academics, but there is little formal control over the debate. In this article, we show that psychologists have tried to solve this issue with various codes of conduct, with an appeal to virtues such as humility, and with practices of self-transformation. We also show that the polemical style of debate, popular in many scientific communities, is itself being questioned by psychologists. Following Shapin and Schaffer’s analysis of the ethics of Robert Boyle’s experimental philosophy in the 17th century, we trace the connections between knowledge, social order, and subjectivity as they are debated and revised by present-day psychologists.

University of Alabama at Birmingham Nathan Shock Center: comparative energetics of aging

The UAB Nathan Shock Center focuses on comparative energetics and aging. Energetics, as defined for this purpose, encompasses the causes, mechanisms, and consequences of the acquisition, storage, and use of metabolizable energy. Comparative energetics is the study of metabolic processes at multiple scales and across multiple species as it relates to health and aging. The link between energetics and aging is increasingly understood in terms of dysregulated mitochondrial function, altered metabolic signaling, and aberrant nutrient responsiveness with increasing age. The center offers world-class expertise in comprehensive, integrated energetic assessment and analysis from the level of the organelle to the organism and across species from the size of worms to rats as well as state-of-the-art data analytics. The range of services offered by our three research cores, (1) The Organismal Energetics Core, (2) Mitometabolism Core, and (3) Data Analytics Core, is described herein.

Predictability in Contemporary Medicine

Medical practice is increasingly coming under the guidance of statistical-mathematical models that are, undoubtedly, valuable tools but are also only a partial representation of reality. Indeed, given that statistics may be more or less adequate, a model is still a subjective interpretation of the researcher and is also influenced by the historical context in which it operates. From this opinion, I will provide a short historical excursus that retraces the advent of probabilistic medicine as a long process that has a beginning that should be sought in the discovery of the complexity of disease. By supporting the belonging of this evolution to the scientific domain it is also acknowledged that the underlying model can be imperfect or fallible and, therefore, confutable as any product of science. Indeed, it seems non-trivial here to recover these concepts, especially today where clinical decisions are entrusted to practical guidelines, which are a hybrid product resulting from the aggregation of multiple perspectives, including the probabilistic approach, to disease. Finally, before the advent of precision medicine, by limiting the use of guidelines to the original consultative context, an aged approach is supported, namely, a relationship with the individual patient.

Updating salamander datasets with phenotypic and stomach content information for two mainland Speleomantes

European plethodontid salamanders (genus Speleomantes; formerly Hydromantes) are a group of eight strictly protected amphibian species which are sensitive to human-induced environmental changes. Long-term monitoring is highly recommended to evaluate their status and to assess potential threats. Here we used two low-impact methodologies to build up a large dataset on two mainland Speleomantes species (S. strinatii and S. ambrosii), which represents an update to two previously published datasets, but also includes several new populations. Specifically, we provide a set of 851 high quality images and a table gathering stomach contents recognized from 560 salamanders. This dataset offers the opportunity to analyse phenotypic traits and stomach contents of eight populations belonging to two Speleomantes species. Furthermore, the data collection performed over different periods allows to expand the potential analyses through a wide temporal scale, allowing long-term studies.

Measurement(s)	Gastric Content • Image
Technology Type(s)	light microscopy • Digital Photography
Sample Characteristic - Organism	Speleomantes • Hydromantes
Sample Characteristic - Environment	Subterranean • karst cave • mine
Sample Characteristic - Location	Italy • Region of Liguria • Municipality of Genova • Province of La Spezia

Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.14346176

Development of a checklist to detect errors in meta-analyses in systematic reviews of interventions: study protocol

Background : Systematic reviews underpin clinical practice and policies that guide healthcare decisions. A core component of many systematic reviews is meta-analysis, which is a statistical synthesis of results across studies. Errors in the conduct and interpretation of meta-analysis can lead to incorrect conclusions regarding the benefits and harms of interventions; and studies have shown that these errors are common. Enabling peer reviewers to better detect errors in meta-analysis through the use of a checklist provides an opportunity for these errors to be rectified before publication. To our knowledge, no such checklist exists. Objective : To develop and evaluate a checklist to detect errors in pairwise meta-analyses in systematic reviews of interventions. Methods : We will undertake a four-step process to develop the checklist. First, we will undertake a systematic review of studies that have evaluated errors in the conduct and interpretation of meta-analysis to generate a bank of items to consider for the checklist. Second, we will undertake a survey of systematic review methodologists and statisticians to seek their views on which items, of the bank of items generated in step 1, are most important to include in the checklist. Third, we will hold a virtual meeting to agree upon which items to include in the checklist. Fourth, before finalising the checklist, we will pilot with editors and peer reviewers of journals. Conclusion : The developed checklist is intended to help journal editors and peer reviewers identify errors in the application and interpretation of meta-analyses in systematic reviews. Fewer errors in the conduct and improved interpretation will lead to more accurate review findings and conclusions to inform clinical practice.

Evidence of misuse of nonparametric tests in the presence of heteroscedasticity within obesity research

BACKGROUND

Classic nonparametric tests (cNPTs), like Kruskal-Wallis or Mann-Whitney U, are sometimes used to detect differences in central tendency ( i.e., means or medians). However, when the tests' assumptions are violated, such as in the presence of unequal variance and other forms of heteroscedasticity, they are no longer valid for testing differences in central tendency. Yet, sometimes researchers erroneously use cNPTs to account for heteroscedasticity.

OBJECTIVE

To document the appropriateness of cNPT use in obesity literature, characterize studies that use cNPTs, and evaluate the citation and public sharing patterns of these articles.

METHODS

We reviewed obesity studies published in 2017 to determine whether the authors used cNPTs: (1) to correct for heteroscedasticity (invalid); (2) when heteroscedasticity was clearly not present (correct); or (3) when it was unclear whether heteroscedasticity was present (unclear). Open science R packages were used to transparently search literature and extract data on how often papers with errors have been cited in academic literature, read in Mendeley, and disseminated in the media.

RESULTS

We identified nine studies that used a cNPT in the presence of heteroscedasticity (some because of the mistaken rationale that the test corrected for heteroscedasticity), 25 articles that did not explicitly state whether heteroscedasticity was present when a cNPT was used, and only four articles that appropriately reported that heteroscedasticity was not present when a cNPT was used. Errors were found in observational and interventional studies, in human and rodent studies, and only when studies were unregistered. Studies with errors have been cited 113 times, read in Mendeley 123 times, and disseminated in the media 41 times, by the public, scientists, science communicators, and doctors.

CONCLUSIONS

Examples of inappropriate use of cNPTs exist in the obesity literature, and those articles perpetuate the errors via various audiences and dissemination platforms.