Research rigor and reproducibility in research education: A CTSA institutional survey

We assessed the rigor and reproducibility (R&R) activities of institutions funded by the National Center for Advancing Translational Sciences (NCTSA) through a survey and website search (N = 61). Of 50 institutional responses, 84% reported incorporating some form of R&R training, 68% reported devoted R&R training, 30% monitored R&R practices, and 10% incentivized them. Website searches revealed 9 (15%) freely available training curricula, and 7 (11%) institutional programs specifically created to enhance R&R. NCATS should formally integrate R&R principles into its translational science models and institutional requirements.

Correcting calculation and data errors reveals that the original conclusions were incorrect in "The best drug supplement for obesity treatment: a systematic review and network meta-analysis"

The goal of this study was to reproduce and evaluate the reliability of the network meta-analysis performed in the article "The best drug supplement for obesity treatment: A systematic review and network meta-analysis" by Salari et al. In recent years, it has become more common to employ network meta-analysis to assess the relative efficacy of treatments often used in clinical practice. To duplicate Salari et al.'s research, we pulled data directly from the original trials and used Cohen's D to determine the effect size for each treatment. We reanalyzed the data since we discovered significant differences between the data we retrieved and the data given by Salari et al. We present new effect size estimates for each therapy and conclude that the prior findings were somewhat erroneous. Our findings highlight the importance of ensuring the accuracy of network meta-analyses to determine the quality and strength of existing evidence.

Is the future of peer review automated?

The rising rate of preprints and publications, combined with persistent inadequate reporting practices and problems with study design and execution, have strained the traditional peer review system. Automated screening tools could potentially enhance peer review by helping authors, journal editors, and reviewers to identify beneficial practices and common problems in preprints or submitted manuscripts. Tools can screen many papers quickly, and may be particularly helpful in assessing compliance with journal policies and with straightforward items in reporting guidelines. However, existing tools cannot understand or interpret the paper in the context of the scientific literature. Tools cannot yet determine whether the methods used are suitable to answer the research question, or whether the data support the authors' conclusions. Editors and peer reviewers are essential for assessing journal fit and the overall quality of a paper, including the experimental design, the soundness of the study's conclusions, potential impact and innovation. Automated screening tools cannot replace peer review, but may aid authors, reviewers, and editors in improving scientific papers. Strategies for responsible use of automated tools in peer review may include setting performance criteria for tools, transparently reporting tool performance and use, and training users to interpret reports.

A System to Easily Manage Metadata in Biomedical Research Labs Based on Open-source Software

In most biomedical labs, researchers gather metadata (i.e., all details about the experimental data) in paper notebooks, spreadsheets, or, sometimes, electronic notebooks. When data analyses occur, the related details usually go into other notebooks or spreadsheets, and more metadata are available. The whole thing rapidly becomes very complex and disjointed, and keeping track of all these things can be daunting. Organizing all the relevant data and related metadata for analysis, publication, sharing, or deposit into archives can be time-consuming, difficult, and prone to errors. By having metadata in a centralized system that contains all details from the start, the process is greatly simplified. While lab management software is available, it can be costly and inflexible. The system described here is based on a popular, freely available, and open-source wiki platform. It provides a simple but powerful way for biomedical research labs to set up a metadata management system linking the whole research process. The system enhances efficiency, transparency, reliability, and rigor, which are key factors to improving reproducibility. The flexibility afforded by the system simplifies implementation of specialized lab requirements and future needs. The protocol presented here describes how to create the system from scratch, how to use it for gathering basic metadata, and provides a fully functional version for perusal by the reader. Graphical abstract: Lab Metadata Management System.

Reflections on Data Sharing Practices in Spinal Cord Injury Research

There are few pharmacological therapeutics available for spinal cord injury despite years of preclinical and clinical research. This brief editorial discusses some of the shortcomings of translational research efforts. In addition, we comment on our previous experiences with data curation and highlight evolving efforts by the spinal cord injury research community to improve prospects for future therapeutic development, especially pertaining to preclinical data sharing through the Open Data Commons for Spinal Cord Injury (ODC-SCI).

Journeying through the DNP project: A qualitative, descriptive study

BACKGROUND

As a requirement of doctor in nursing practice (DNP) programs, a final scholarly project is required. Little is known about the student experience initiating, implementing, evaluating, and disseminating the scholarly DNP project.

PURPOSE

The purpose of this qualitative, descriptive study was to explore descriptions of what it is like to move through the DNP project process, from the perspective of successful recent DNP graduates.

METHOD

Using purposive, convenience, and snowball sampling, 15 recent DNP graduates were recruited to participate in a semi-structured interview. Using a pragmatic, open coding approach with constant comparison, four researchers evaluated the transcripts and reduced the data three times by coding and categorizing, clustering responses so the essential theme emerged.

RESULTS

Nine categories were interpreted, and one overarching theme emerged: Journeying Through the DNP Project. A metaphor weaving together the categories is offered.

CONCLUSION

Faculty support, through communication and mentorship, is strongly encouraged. Rigor of projects needs to be enhanced. Student experiences may be improved by faculty establishing supportive student relationships and ensuring that faculty understand the differences between and among research, evidence-based practice, quality improvement, and process improvement. The DNP student experience can be enhanced with program support and faculty mentorship and support.

The Health Sciences and Technology Academy (HSTA): Providing 26 Years of Academic and Social Support to Appalachian Youth in West Virginia

The Health Sciences and Technology Academy's, (HSTA) goals are to increase college attendance of African American, financially disadvantaged, first generation college and rural Appalachian youth and increase health-care providers and STEM professionals in underserved communities. Students enter in the 9th grade and remain in HSTA four years. They engage in a rigorous academic program within the nurturing environment of small after-school clubs punctuated by yearly summer camps on multiple college campuses. A distinctive piece of HSTA is its students' development of research projects under the mentorship of teachers and researchers that examine and address health issues faced by their communities. The projects help HSTA students to understand the health dynamics in their local community, transforming them into community advocates who address health and social issues at home as they prepare to move on to college and beyond. Substantial in-state tuition waivers inspire 99% of the 3,021 HSTA graduates to attend college versus 56% of WV high school graduates. Approximately 85% of matriculating HSTA students graduate with a four-year degree or higher versus less than 50% of all college entrants. To date, 57% of HSTA students go into health and other STEM majors, much higher than the state and national figures.

A framework for clinical and translational research in the era of rigor and reproducibility

INTRODUCTION

Rigor and reproducibility are two important cornerstones of medical and scientific advancement. Clinical and translational research (CTR) contains four phases (T1-T4), involving the translation of basic research to humans, then to clinical settings, practice, and the population, with the ultimate goal of improving public health. Here we provide a framework for rigorous and reproducible CTR.

METHODS

In this paper we define CTR, provide general and phase-specific recommendations for improving quality and reproducibility of CTR with emphases on study design, data collection and management, analyses and reporting. We present and discuss aspects of rigor and reproducibility following published examples of CTR from the literature, including one example that shows the development path of different treatments that address anaplastic lymphoma kinase-positive (ALK+) non-small cell lung cancer (NSCLC).

RESULTS

It is particularly important to consider robust and unbiased experimental design and methodology for analysis and interpretation for clinical translation studies to ensure reproducibility before taking the next translational step. There are both commonality and differences along the clinical translation research phases in terms of research focuses and considerations regarding study design, implementation, and data analysis approaches.

CONCLUSIONS

Sound scientific practices, starting with rigorous study design, transparency, and team efforts can greatly enhance CTR. Investigators from multidisciplinary teams should work along the spectrum of CTR phases, and identify optimal practices for study design, data collection, data analysis, and results reporting to allow timely advances in the relevant field of research.

Rigor and reproducibility in rodent behavioral research

Behavioral neuroscience research incorporates the identical high level of meticulous methodologies and exacting attention to detail as all other scientific disciplines. To achieve maximal rigor and reproducibility of findings, well-trained investigators employ a variety of established best practices. Here we explicate some of the requirements for rigorous experimental design and accurate data analysis in conducting mouse and rat behavioral tests. Novel object recognition is used as an example of a cognitive assay which has been conducted successfully with a range of methods, all based on common principles of appropriate procedures, controls, and statistics. Directors of Rodent Core facilities within Intellectual and Developmental Disabilities Research Centers contribute key aspects of their own novel object recognition protocols, offering insights into essential similarities and less-critical differences. Literature cited in this review article will lead the interested reader to source papers that provide step-by-step protocols which illustrate optimized methods for many standard rodent behavioral assays. Adhering to best practices in behavioral neuroscience will enhance the value of animal models for the multiple goals of understanding biological mechanisms, evaluating consequences of genetic mutations, and discovering efficacious therapeutics.

Common data elements (CDEs) for preclinical epilepsy research: Introduction to CDEs and description of core CDEs. A TASK3 report of the ILAE/AES joint translational task force

Common data elements (CDEs) are becoming more common as more areas of preclinical research have generated CDEs. Herein we provide an overview of the progress to date in generating CDEs for preclinical epilepsy research. Currently there are CDEs that have been developed for Physiology (in vivo), Behavior, Pharmacology, and Electroencephalography (EEG). Together the CDEs and methodologic considerations associated with these CDEs are laid out in consecutive manuscripts published in Epilepsia Open, each describing CDEs for their respective topic area. In addition to the overview of progress for the 4 subjects, core characteristics (Core CDEs) are described and explained. Data collection using a case report form (CRF) is described, and considerations that are involved in using the CDEs and CRFs are discussed.

Introduction: Scientific rigor in paleopathology

This introductory chapter to the Special Issue on "Scientific Rigor in Paleopathology" serves to orient and introduce the chapters that follow through a detailed consideration of paleopathology as a 21^st century intellectual field. In this vein, we first make the significant point that paleopathology is a profoundly interdisciplinary endeavor, encompassing aspects of the biomedical science, the humanities, and the social sciences. Thus, we suggest that no one practitioner can personally command the range of skills necessary for a 21^st century paleopathologist. To maintain rigor in differential diagnosis, we emphasize collaborations and consider key concepts that illustrate the basic knowledge from each of these fields that any paleopathologist should command. We then address the manner in which disease diagnosis should proceed as a scientific endeavor. To illustrate scientific rigor in differential diagnosis, we present two case studies drawn from 1970s contributions by Cook and by Buikstra. Finally, we introduce Chapters 2-6, which address differential diagnosis in contexts ranging from specific conditions (scurvy, trepanation) to broader field-wide considerations (paleoparasitology, historical paleopathology, imaging, animal paleopathology).

Applying the new SABV (sex as a biological variable) policy to research and clinical care

Sex as a biological variable (SABV) is a key part of the new National Institutes of Health (NIH) initiative to enhance reproducibility through rigor and transparency. The SABV policy requires researchers to factor sex into the design, analysis, and reporting of vertebrate animal and human studies. The policy was implemented as it has become increasingly clear that male/female differences extend well beyond reproductive and hormonal issues. Implementation of the policy is also meant to address inattention to sex influences in biomedical research. Sex affects: cell physiology, metabolism, and many other biological functions; symptoms and manifestations of disease; and responses to treatment. For example, sex has profound influences in neuroscience, from circuitry to physiology to pain perception. Extending beyond the robust efforts of NIH to ensure that women are included in clinical trials, the SABV policy also includes rigorous preclinical experimental designs that inform clinical research. Additionally, the NIH has engaged journal editors and publishers to facilitate reproducibility by addressing rigor and promoting transparency through scientifically appropriate sex-specific study results reporting. The Sex And Gender Equity in Research (SAGER) guidelines were developed to assist researchers and journal editors in reporting sex and gender information in publications [1].

Some Thoughts on Experimental Design

Perhaps even more important than the techniques themselves are the quality of the biological questions asked and the design of the experiments devised to answer them. This chapter summarizes some of the key issues and also touches on how the same principles affect scholarly use of the scientific literature and good peer-reviewing practices.