1
|
Lan M, Cheng M, Hoang L, Ter Riet G, Kilicoglu H. Automatic categorization of self-acknowledged limitations in randomized controlled trial publications. J Biomed Inform 2024; 152:104628. [PMID: 38548008 DOI: 10.1016/j.jbi.2024.104628] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2023] [Revised: 03/09/2024] [Accepted: 03/24/2024] [Indexed: 04/05/2024]
Abstract
OBJECTIVE Acknowledging study limitations in a scientific publication is a crucial element in scientific transparency and progress. However, limitation reporting is often inadequate. Natural language processing (NLP) methods could support automated reporting checks, improving research transparency. In this study, our objective was to develop a dataset and NLP methods to detect and categorize self-acknowledged limitations (e.g., sample size, blinding) reported in randomized controlled trial (RCT) publications. METHODS We created a data model of limitation types in RCT studies and annotated a corpus of 200 full-text RCT publications using this data model. We fine-tuned BERT-based sentence classification models to recognize the limitation sentences and their types. To address the small size of the annotated corpus, we experimented with data augmentation approaches, including Easy Data Augmentation (EDA) and Prompt-Based Data Augmentation (PromDA). We applied the best-performing model to a set of about 12K RCT publications to characterize self-acknowledged limitations at larger scale. RESULTS Our data model consists of 15 categories and 24 sub-categories (e.g., Population and its sub-category DiagnosticCriteria). We annotated 1090 instances of limitation types in 952 sentences (4.8 limitation sentences and 5.5 limitation types per article). A fine-tuned PubMedBERT model for limitation sentence classification improved upon our earlier model by about 1.5 absolute percentage points in F1 score (0.821 vs. 0.8) with statistical significance (p<.001). Our best-performing limitation type classification model, PubMedBERT fine-tuning with PromDA (Output View), achieved an F1 score of 0.7, improving upon the vanilla PubMedBERT model by 2.7 percentage points, with statistical significance (p<.001). CONCLUSION The model could support automated screening tools which can be used by journals to draw the authors' attention to reporting issues. Automatic extraction of limitations from RCT publications could benefit peer review and evidence synthesis, and support advanced methods to search and aggregate the evidence from the clinical trial literature.
Collapse
Affiliation(s)
- Mengfei Lan
- School of Information Sciences, University of Illinois Urbana-Champaign, 501 Daniel Street, Champaign, 61820, IL, USA
| | - Mandy Cheng
- Department of Biological Sciences, Binghamton University, 4400 Vestal Parkway East, New York City, 13902, NY, USA
| | - Linh Hoang
- School of Information Sciences, University of Illinois Urbana-Champaign, 501 Daniel Street, Champaign, 61820, IL, USA
| | - Gerben Ter Riet
- Faculty of Health, Amsterdam University of Applied Sciences, Tafelbergweg 51, Amsterdam, 1105 BD, The Netherlands
| | - Halil Kilicoglu
- School of Information Sciences, University of Illinois Urbana-Champaign, 501 Daniel Street, Champaign, 61820, IL, USA.
| |
Collapse
|
2
|
Hankenson FC, Prager EM, Berridge BR. Advocating for Generalizability: Accepting Inherent Variability in Translation of Animal Research Outcomes. Annu Rev Anim Biosci 2024; 12:391-410. [PMID: 38358839 DOI: 10.1146/annurev-animal-021022-043531] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/17/2024]
Abstract
Advancing scientific discovery requires investigators to embrace research practices that increase transparency and disclosure about materials, methods, and outcomes. Several research advocacy and funding organizations have produced guidelines and recommended practices to enhance reproducibility through detailed and rigorous research approaches; however, confusion around vocabulary terms and a lack of adoption of suggested practices have stymied successful implementation. Although reproducibility of research findings cannot be guaranteed due to extensive inherent variables in attempts at experimental repetition, the scientific community can advocate for generalizability in the application of data outcomes to ensure a broad and effective impact on the comparison of animals to translation within human research. This report reviews suggestions, based upon work with National Institutes of Health advisory groups, for improving rigor and transparency in animal research through aspects of experimental design, statistical assessment, and reporting factors to advocate for generalizability in the application of comparative outcomes between animals and humans.
Collapse
Affiliation(s)
- F C Hankenson
- Division of Laboratory Animal Medicine, Department of Pathobiology, School of Veterinary Medicine and University Laboratory Animal Resources, University of Pennsylvania, Philadelphia, Pennsylvania, USA;
| | - E M Prager
- Research Program Management, Regeneron Pharmaceuticals, Inc., Tarrytown, New York, USA;
| | - B R Berridge
- B2 Pathology Solutions LLC, Cary, North Carolina, USA;
| |
Collapse
|
3
|
Axfors C, Malički M, Goodman SN. Research rigor and reproducibility in research education: A CTSA institutional survey. J Clin Transl Sci 2024; 8:e45. [PMID: 38476247 PMCID: PMC10928701 DOI: 10.1017/cts.2024.10] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2023] [Revised: 01/05/2024] [Accepted: 01/09/2024] [Indexed: 03/14/2024] Open
Abstract
We assessed the rigor and reproducibility (R&R) activities of institutions funded by the National Center for Advancing Translational Sciences (NCTSA) through a survey and website search (N = 61). Of 50 institutional responses, 84% reported incorporating some form of R&R training, 68% reported devoted R&R training, 30% monitored R&R practices, and 10% incentivized them. Website searches revealed 9 (15%) freely available training curricula, and 7 (11%) institutional programs specifically created to enhance R&R. NCATS should formally integrate R&R principles into its translational science models and institutional requirements.
Collapse
Affiliation(s)
- Cathrine Axfors
- Stanford University School of Medicine,
Stanford Program on Research Rigor & Reproducibility (SPORR), Stanford,
CA, USA
- Meta-Research Innovation Center at Stanford (METRICS),
Stanford University, Stanford, CA,
USA
| | - Mario Malički
- Stanford University School of Medicine,
Stanford Program on Research Rigor & Reproducibility (SPORR), Stanford,
CA, USA
- Meta-Research Innovation Center at Stanford (METRICS),
Stanford University, Stanford, CA,
USA
- Department of Epidemiology and Population Health, Stanford
University School of Medicine, Stanford, CA,
USA
| | - Steven N. Goodman
- Stanford University School of Medicine,
Stanford Program on Research Rigor & Reproducibility (SPORR), Stanford,
CA, USA
- Meta-Research Innovation Center at Stanford (METRICS),
Stanford University, Stanford, CA,
USA
- Department of Epidemiology and Population Health, Stanford
University School of Medicine, Stanford, CA,
USA
- Department of Medicine, Stanford University School of
Medicine, Stanford, CA, USA
| |
Collapse
|
4
|
Piekniewska A, Anderson N, Roelandse M, Lloyd KCK, Korf I, Voss SR, de Castro G, Magnani DM, Varga Z, James-Zorn C, Horb M, Grethe JS, Bandrowski A. Do organisms need an impact factor? Citations of key biological resources including model organisms reveal usage patterns and impact. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.01.15.575636. [PMID: 38293091 PMCID: PMC10827057 DOI: 10.1101/2024.01.15.575636] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/01/2024]
Abstract
Research resources like transgenic animals and antibodies are the workhorses of biomedicine, enabling investigators to relatively easily study specific disease conditions. As key biological resources, transgenic animals and antibodies are often validated, maintained, and distributed from university based stock centers. As these centers heavily rely largely on grant funding, it is critical that they are cited by investigators so that usage can be tracked. However, unlike systems for tracking the impact of papers, the conventions and systems for tracking key resource usage and impact lag behind. Previous studies have shown that about 50% of the resources are not findable, making the studies they are supporting irreproducible, but also makes tracking resources difficult. The RRID project is filling this gap by working with journals and resource providers to improve citation practices and to track the usage of these key resources. Here, we reviewed 10 years of citation practices for five university based stock centers, characterizing each reference into two broad categories: findable (authors could use the RRID, stock number, or full name) and not findable (authors could use a nickname or a common name that is not unique to the resource). The data revealed that when stock centers asked their communities to cite resources by RRID, in addition to helping stock centers more easily track resource usage by increasing the number of RRID papers, authors shifted from citing resources predominantly by nickname (~50% of the time) to citing them by one of the findable categories (~85%) in a matter of several years. In the case of one stock center, the MMRRC, the improvement in findability is also associated with improvements in the adherence to NIH rigor criteria, as determined by a significant increase in the Rigor and Transparency Index for studies using MMRRC mice. From this data, it was not possible to determine whether outreach to authors or changes to stock center websites drove better citation practices, but findability of research resources and rigor adherence was improved.
Collapse
Affiliation(s)
| | | | | | - K C Kent Lloyd
- Mouse Biology Program, Comprehensive Cancer Center, and Department of Surgery, School of Medicine, University of California, Davis
| | - Ian Korf
- University of California Davis, Department of Molecular and Cellular Biology; UC Davis Genome Center
| | - S Randal Voss
- Ambystoma Genetic Stock Center, Spinal Cord and Brain Injury Research Center, University of Kentucky
| | | | | | - Zoltan Varga
- Zebrafish International Resource Center, Institute of Neuroscience, University of Oregon
| | - Christina James-Zorn
- Cincinnati Children's Research Foundation, Division of Developmental Biology, www.Xenbase.org
| | - Marko Horb
- National Xenopus Resource, Eugene Bell Center for Regenerative Biology and Tissue Engineering, Marine Biological Laboratory
| | - Jeffery S Grethe
- University of California at San Diego, School of Medicine, Department of Neuroscience
| | - Anita Bandrowski
- University of California at San Diego, Department of Neuroscience; SciCrunch Inc
| |
Collapse
|
5
|
LaFlamme M, Harney J, Hrynaszkiewicz I. A survey of researchers' methods sharing practices and priorities. PeerJ 2024; 12:e16731. [PMID: 38188149 PMCID: PMC10771089 DOI: 10.7717/peerj.16731] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Accepted: 12/06/2023] [Indexed: 01/09/2024] Open
Abstract
Missing or inaccessible information about the methods used in scientific research slows the pace of discovery and hampers reproducibility. Yet little is known about how, why, and under what conditions researchers share detailed methods information, or about how such practices vary across social categories like career stage, field, and region. In this exploratory study, we surveyed 997 active researchers about their attitudes and behaviors with respect to methods sharing. The most common approach reported by respondents was private sharing upon request, but a substantial minority (33%) had publicly shared detailed methods information independently of their research findings. The most widely used channels for public sharing were connected to peer-reviewed publications, while the most significant barriers to public sharing were found to be lack of time and lack of awareness about how or where to share. Insofar as respondents were moderately satisfied with their ability to accomplish various goals associated with methods sharing, we conclude that efforts to increase public sharing may wish to focus on enhancing and building awareness of existing solutions-even as future research should seek to understand the needs of methods users and the extent to which they align with prevailing practices of sharing.
Collapse
Affiliation(s)
| | - James Harney
- Public Library of Science, San Francisco, CA, United States
| | | |
Collapse
|
6
|
Plante J, Langerwerf L, Klopper M, Rhon DI, Young JL. Evaluation of Transparency and Openness Guidelines in Physical Therapist Journals. Phys Ther 2024; 104:pzad133. [PMID: 37815940 DOI: 10.1093/ptj/pzad133] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Revised: 06/05/2023] [Accepted: 08/21/2023] [Indexed: 10/12/2023]
Abstract
OBJECTIVE The goals of this study were to evaluate the extent that physical therapist journals support open science research practices by adhering to the Transparency and Openness Promotion (TOP) guidelines and to assess the relationship between journal scores and their respective journal impact factor (JIF). METHODS Scimago, mapping studies, the National Library of Medicine, and journal author guidelines were searched to identify physical therapist journals for inclusion. Journals were graded on 10 standards (29 available total points) related to transparency with data, code, research materials, study design and analysis, preregistration of studies and statistical analyses, replication, and open science badges. The relationship between journal transparency and openness scores and their JIF was determined. RESULTS Thirty-five journals' author guidelines were assigned transparency and openness factor scores. The median score (interquartile range) across journals was 3.00 out of 29 (3.00) points (for all journals the scores ranged from 0 to 8). The 2 standards with the highest degree of implementation were design and analysis transparency (reporting guidelines) and study preregistration. No journals reported on code transparency, materials transparency, replication, and open science badges. TOP factor scores were a significant predictor of JIF scores. CONCLUSION There is low implementation of the TOP standards by physical therapist journals. TOP factor scores demonstrated predictive abilities for JIF scores. Policies from journals must improve to make open science practices the standard in research. Journals are in an influential position to guide practices that can improve the rigor of publication which, ultimately, enhances the evidence-based information used by physical therapists. IMPACT Transparent, open, and reproducible research will move the profession forward by improving the quality of research and increasing the confidence in results for implementation in clinical care.
Collapse
Affiliation(s)
- Jacqueline Plante
- Department of Physical Therapy, Doctor of Science in Physical Therapy Program, Bellin College, Green Bay, Wisconsin, USA
| | - Leigh Langerwerf
- Department of Physical Therapy, Doctor of Science in Physical Therapy Program, Bellin College, Green Bay, Wisconsin, USA
| | - Mareli Klopper
- Department of Physical Therapy, Doctor of Science in Physical Therapy Program, Bellin College, Green Bay, Wisconsin, USA
| | - Daniel I Rhon
- Department of Physical Therapy, Doctor of Science in Physical Therapy Program, Bellin College, Green Bay, Wisconsin, USA
- Department of Rehabilitation Medicine, School of Medicine, Uniformed Services University of the Health Sciences, Bethesda, Maryland, USA
| | - Jodi L Young
- Department of Physical Therapy, Doctor of Science in Physical Therapy Program, Bellin College, Green Bay, Wisconsin, USA
| |
Collapse
|
7
|
Thibault RT, Amaral OB, Argolo F, Bandrowski AE, Davidson AR, Drude NI. Open Science 2.0: Towards a truly collaborative research ecosystem. PLoS Biol 2023; 21:e3002362. [PMID: 37856538 PMCID: PMC10617723 DOI: 10.1371/journal.pbio.3002362] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Revised: 10/31/2023] [Indexed: 10/21/2023] Open
Abstract
Conversations about open science have reached the mainstream, yet many open science practices such as data sharing remain uncommon. Our efforts towards openness therefore need to increase in scale and aim for a more ambitious target. We need an ecosystem not only where research outputs are openly shared but also in which transparency permeates the research process from the start and lends itself to more rigorous and collaborative research. To support this vision, this Essay provides an overview of a selection of open science initiatives from the past 2 decades, focusing on methods transparency, scholarly communication, team science, and research culture, and speculates about what the future of open science could look like. It then draws on these examples to provide recommendations for how funders, institutions, journals, regulators, and other stakeholders can create an environment that is ripe for improvement.
Collapse
Affiliation(s)
- Robert T. Thibault
- 1 Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, Unites States of America
| | - Olavo B. Amaral
- Institute of Medical Biochemistry Leopoldo de Meis, Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil
| | | | - Anita E. Bandrowski
- FAIR Data Informatics Lab, Department of Neuroscience, UCSD, San Diego, California, United States of America
- SciCrunch Inc., San Diego, California, United States of America
| | - Alexandra R, Davidson
- Institute for Evidence-Based Health Care, Bond University, Robina, Australia
- Faculty of Health Science and Medicine, Bond University, Robina, Australia
| | - Natascha I. Drude
- Berlin Institute of Health (BIH) at Charité, BIH QUEST Center for Responsible Research, Berlin, Germany
| |
Collapse
|
8
|
Kilicoglu H, Jiang L, Hoang L, Mayo-Wilson E, Vinkers CH, Otte WM. Methodology reporting improved over time in 176,469 randomized controlled trials. J Clin Epidemiol 2023; 162:19-28. [PMID: 37562729 PMCID: PMC10829891 DOI: 10.1016/j.jclinepi.2023.08.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2023] [Revised: 07/25/2023] [Accepted: 08/02/2023] [Indexed: 08/12/2023]
Abstract
OBJECTIVES To describe randomized controlled trial (RCT) methodology reporting over time. STUDY DESIGN AND SETTING We used a deep learning-based sentence classification model based on the Consolidated Standards of Reporting Trials (CONSORT) statement, considered minimum requirements for reporting RCTs. We included 176,469 RCT reports published between 1966 and 2018. We analyzed the reporting trends over 5-year time periods, grouping trials from 1966 to 1990 in a single stratum. We also explored the effect of journal impact factor (JIF) and medical discipline. RESULTS Population, Intervention, Comparator, Outcome (PICO) items were commonly reported during each period, and reporting increased over time (e.g., interventions: 79.1% during 1966-1990 to 87.5% during 2010-2018). Reporting of some methods information has increased, although there is room for improvement (e.g., sequence generation: 10.8-41.8%). Some items are reported infrequently (e.g., allocation concealment: 5.1-19.3%). The number of items reported and JIF are weakly correlated (Pearson's r (162,702) = 0.16, P < 0.001). The differences in the proportion of items reported between disciplines are small (<10%). CONCLUSION Our analysis provides large-scale quantitative support for the hypothesis that RCT methodology reporting has improved over time. Extending these models to all CONSORT items could facilitate compliance checking during manuscript authoring and peer review, and support metaresearch.
Collapse
Affiliation(s)
- Halil Kilicoglu
- School of Information Sciences, University of Illinois Urbana-Champaign, Champaign, IL, USA.
| | - Lan Jiang
- School of Information Sciences, University of Illinois Urbana-Champaign, Champaign, IL, USA
| | - Linh Hoang
- School of Information Sciences, University of Illinois Urbana-Champaign, Champaign, IL, USA
| | - Evan Mayo-Wilson
- Department of Epidemiology, University of North Carolina School of Global Public Health, Chapel Hill, NC, USA
| | - Christiaan H Vinkers
- Department of Psychiatry and Anatomy & Neurosciences, Amsterdam University Medical Center Location Vrije Universiteit Amsterdam, 1081 HV, Amsterdam, The Netherlands; Amsterdam Public Health, Mental Health Program and Amsterdam Neuroscience, Mood, Anxiety, Psychosis, Sleep & Stress Program, Amsterdam, The Netherlands; GGZ inGeest Mental Health Care, 1081 HJ, Amsterdam, The Netherlands
| | - Willem M Otte
- Department of Child Neurology, UMC Utrecht Brain Center, University Medical Center Utrecht, and Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
9
|
Crawford DC, Hoye ML, Silberberg SD. From Methods to Monographs: Fostering a Culture of Research Quality. eNeuro 2023; 10:ENEURO.0247-23.2023. [PMID: 37553250 PMCID: PMC10411680 DOI: 10.1523/eneuro.0247-23.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Accepted: 07/20/2023] [Indexed: 08/10/2023] Open
Affiliation(s)
- Devon C Crawford
- Office of Research Quality, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD 20892
| | - Mariah L Hoye
- Office of Research Quality, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD 20892
| | - Shai D Silberberg
- Office of Research Quality, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD 20892
| |
Collapse
|
10
|
Zurrer WE, Cannon AE, Ewing E, Rosso M, Reich DS, Ineichen BV. Auto-STEED: A data mining tool for automated extraction of experimental parameters and risk of bias items from in vivo publications. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.02.24.529867. [PMID: 37205453 PMCID: PMC10187249 DOI: 10.1101/2023.02.24.529867] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
Background Systematic reviews, i.e., research summaries that address focused questions in a structured and reproducible manner, are a cornerstone of evidence-based medicine and research. However, certain systematic review steps such as data extraction are labour-intensive which hampers their applicability, not least with the rapidly expanding body of biomedical literature. Objective To bridge this gap, we aimed at developing a data mining tool in the R programming environment to automate data extraction from neuroscience in vivo publications. The function was trained on a literature corpus (n=45 publications) of animal motor neuron disease studies and tested in two validation corpora (motor neuron diseases, n=31 publications; multiple sclerosis, n=244 publications). Results Our data mining tool Auto-STEED (Automated and STructured Extraction of Experimental Data) was able to extract key experimental parameters such as animal models and species as well as risk of bias items such as randomization or blinding from in vivo studies. Sensitivity and specificity were over 85 and 80%, respectively, for most items in both validation corpora. Accuracy and F-scores were above 90% and 0.9 for most items in the validation corpora. Time savings were above 99%. Conclusions Our developed text mining tool Auto-STEED is able to extract key experimental parameters and risk of bias items from the neuroscience in vivo literature. With this, the tool can be deployed to probe a field in a research improvement context or to replace one human reader during data extraction resulting in substantial time-savings and contribute towards automation of systematic reviews. The function is available on Github.
Collapse
|
11
|
Du X, Dastmalchi F, Ye H, Garrett TJ, Diller MA, Liu M, Hogan WR, Brochhausen M, Lemas DJ. Evaluating LC-HRMS metabolomics data processing software using FAIR principles for research software. Metabolomics 2023; 19:11. [PMID: 36745241 DOI: 10.1007/s11306-023-01974-3] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/08/2022] [Accepted: 01/20/2023] [Indexed: 02/07/2023]
Abstract
BACKGROUND Liquid chromatography-high resolution mass spectrometry (LC-HRMS) is a popular approach for metabolomics data acquisition and requires many data processing software tools. The FAIR Principles - Findability, Accessibility, Interoperability, and Reusability - were proposed to promote open science and reusable data management, and to maximize the benefit obtained from contemporary and formal scholarly digital publishing. More recently, the FAIR principles were extended to include Research Software (FAIR4RS). AIM OF REVIEW This study facilitates open science in metabolomics by providing an implementation solution for adopting FAIR4RS in the LC-HRMS metabolomics data processing software. We believe our evaluation guidelines and results can help improve the FAIRness of research software. KEY SCIENTIFIC CONCEPTS OF REVIEW We evaluated 124 LC-HRMS metabolomics data processing software obtained from a systematic review and selected 61 software for detailed evaluation using FAIR4RS-related criteria, which were extracted from the literature along with internal discussions. We assigned each criterion one or more FAIR4RS categories through discussion. The minimum, median, and maximum percentages of criteria fulfillment of software were 21.6%, 47.7%, and 71.8%. Statistical analysis revealed no significant improvement in FAIRness over time. We identified four criteria covering multiple FAIR4RS categories but had a low %fulfillment: (1) No software had semantic annotation of key information; (2) only 6.3% of evaluated software were registered to Zenodo and received DOIs; (3) only 14.5% of selected software had official software containerization or virtual machine; (4) only 16.7% of evaluated software had a fully documented functions in code. According to the results, we discussed improvement strategies and future directions.
Collapse
Affiliation(s)
- Xinsong Du
- Department of Health Outcomes and Biomedical Informatics, University of Florida College of Medicine, Gainesville, FL, USA
| | - Farhad Dastmalchi
- Department of Health Outcomes and Biomedical Informatics, University of Florida College of Medicine, Gainesville, FL, USA
| | - Hao Ye
- Health Science Center Libraries, University of Florida, Florida, USA
| | - Timothy J Garrett
- Department of Pathology, Immunology and Laboratory Medicine, College of Medicine, University of Florida, Florida, USA
| | - Matthew A Diller
- Department of Health Outcomes and Biomedical Informatics, University of Florida College of Medicine, Gainesville, FL, USA
| | - Mei Liu
- Department of Health Outcomes and Biomedical Informatics, University of Florida College of Medicine, Gainesville, FL, USA
| | - William R Hogan
- Department of Health Outcomes and Biomedical Informatics, University of Florida College of Medicine, Gainesville, FL, USA
| | - Mathias Brochhausen
- Department of Biomedical Informatics, College of Medicine, University of Arkansas for Medical Sciences, Little Rock, USA
| | - Dominick J Lemas
- Department of Health Outcomes and Biomedical Informatics, University of Florida College of Medicine, Gainesville, FL, USA.
- Department of Obstetrics and Gynecology, University of Florida College of Medicine, Florida, Gainesville, United States.
- Center for Perinatal Outcomes Research, University of Florida College of Medicine, Gainesville, United States.
| |
Collapse
|
12
|
Alnaimat F, Sweis NJ, Sweis JJG, Ascoli C, Korsten P, Rubinstein I, Sweiss NJ. Reproducibility and rigor in rheumatology research. Front Med (Lausanne) 2023; 9:1073551. [PMID: 36687429 PMCID: PMC9853178 DOI: 10.3389/fmed.2022.1073551] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2022] [Accepted: 12/15/2022] [Indexed: 01/09/2023] Open
Abstract
The pillars of scientific progress in rheumatology are experimentation and observation, followed by the publication of reliable and credible results. These data must then be independently verified, validated, and replicated. Peer and journal-specific technical and statistical reviews are paramount to improving rigor and reproducibility. In addition, research integrity, ethics, and responsible conduct training can help to reduce research misconduct and improve scientific evidence. As the number of published articles in rheumatology grows, the field has become critical for determining reproducibility. Prospective, longitudinal, randomized controlled clinical trials are the gold standard for evaluating clinical intervention efficacy and safety in this space. However, their applicability to larger, more representative patient populations with rheumatological disorders worldwide could be limited due to time, technical, and cost constraints involved with large-scale clinical trials. Accordingly, analysis of real-world, patient-centered clinical data retrieved from established healthcare inventories, such as electronic health records, medical billing reports, and disease registries, are increasingly used to report patient outcomes. Unfortunately, it is unknown whether this clinical research paradigm in rheumatology could be deployed in medically underserved regions.
Collapse
Affiliation(s)
- Fatima Alnaimat
- Division of Rheumatology, Department of Medicine, The University of Jordan, Amman, Jordan
| | - Nadia J. Sweis
- Department of Business Administration, King Talal School of Business Technology, Princess Sumaya University for Technology, Amman, Jordan
| | | | - Christian Ascoli
- Division of Pulmonary, Critical Care, Sleep, and Allergy, Department of Medicine, University of Illinois Chicago, Chicago, IL, United States
| | - Peter Korsten
- Department of Nephrology and Rheumatology, University Medical Center Göttingen, Göttingen, Germany
| | - Israel Rubinstein
- Division of Pulmonary, Critical Care, Sleep, and Allergy, Department of Medicine, University of Illinois Chicago, Chicago, IL, United States
| | - Nadera J. Sweiss
- Division of Rheumatology, Department of Medicine, University of Illinois Chicago, Chicago, IL, United States
| |
Collapse
|
13
|
Karp NA, Pearl EJ, Stringer EJ, Barkus C, Ulrichsen JC, Percie du Sert N. A qualitative study of the barriers to using blinding in in vivo experiments and suggestions for improvement. PLoS Biol 2022; 20:e3001873. [PMID: 36395326 PMCID: PMC9714947 DOI: 10.1371/journal.pbio.3001873] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2021] [Revised: 12/01/2022] [Accepted: 10/07/2022] [Indexed: 11/18/2022] Open
Abstract
In animal experiments, blinding (also known as masking) is a methodological strategy to reduce the risk that scientists, animal care staff, or other staff involved in the research may consciously or subconsciously influence the outcome. Lack of masking has been shown to correlate with an overestimation of treatment efficacy and false positive findings. We conducted exploratory interviews across academic and a commercial setting to discuss the implementation of masking at four stages of the experiment: during allocation and intervention, during the conduct of the experiment, during the outcome assessment, and during the data analysis. The objective was to explore the awareness, engagement, perceptions, and the barriers to implementing masking in animal experiments. We conducted multiple interviews, to explore 30 different experiments, and found examples of excellent practice but also areas where masking was rarely implemented. Significant barriers arose from the operational and informatic systems implemented. These systems have prioritised the management of welfare without considering how to allow researchers to use masking in their experiments. For some experiments, there was a conflict between the management of welfare for an individual animal versus delivering a robust experiment where all animals are treated in the same manner. We identified other challenges related to the level of knowledge on the purpose of masking or the implementation and the work culture. The exploration of these issues provides insight into how we, as a community, can identify the most significant barriers in a given research environment. Here, we offer practical solutions to enable researchers to implement masking as standard. To move forward, we need both the individual scientists to embrace the use of masking and the facility managers and institutes to engage and provide a framework that supports the scientists.
Collapse
Affiliation(s)
- Natasha A. Karp
- Data Sciences & Quantitative Biology, Discovery Sciences, R&D, AstraZeneca, Cambridge, United Kingdom
- * E-mail:
| | | | - Emma J. Stringer
- Biomedical Services Unit, University of Birmingham, Birmingham, United Kingdom
| | | | | | | |
Collapse
|
14
|
Chiriboga L, Callis GM, Wang Y, Chlipala E. Guide for collecting and reporting metadata on protocol variables and parameters from slide-based histotechnology assays to enhance reproducibility. J Histotechnol 2022; 45:132-147. [DOI: 10.1080/01478885.2022.2134022] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Affiliation(s)
- Luis Chiriboga
- Department of Pathology, NYU Grossman School of Medicine, New York, NY, USA
- NYULH Center for Biospecimen Research and Development, New York, NY, USA
| | | | - Yongfu Wang
- Stowers Institute for Medical Research, Kansas, MO, USA
| | | |
Collapse
|
15
|
Ott AW, Sol-Church K, Deshpande GM, Knudtson KL, Meyn SM, Mische SM, Taatjes DJ, Sturges MR, Gregory CW. Rigor, Reproducibility, and Transparency in Shared Research Resources: Follow-Up Survey and Recommendations for Improvements. J Biomol Tech 2022; 33:3fc1f5fe.fa789303. [PMID: 36910580 PMCID: PMC10001929 DOI: 10.7171/3fc1f5fe.fa789303] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
Abstract
Rigor, reproducibility, and transparency (RR&T) are essential components of all scientific pursuits. Shared research resources, also known as core facilities, are on the frontlines of ensuring robust RR&T practices. The Association of Biomolecular Resource Facilities Committee on Core Rigor and Reproducibility conducted a follow-up survey 4 years after the initial 2017 survey to determine if core facilities have seen a positive impact of new RR&T initiatives (including guidance from the National Institutes of Health, new scientific journal requirements on transparency and data provenance, and educational tools from professional organizations). While there were fewer participants in the most recent survey, the respondents' opinions on the role of core facilities and level of best practices adoption remained the same. Overall, the respondents agreed that procedures should be implemented by core facilities to ensure scientific RR&T. They also indicated that there is a strong correlation between institutions that emphasize RR&T and core customers using this expertise in grant applications and publications. The survey also assessed the impact of the COVID-19 pandemic on core operations and RR&T. The answers to these pandemic-related questions revealed that many of the strategies aimed at increasing efficiencies are also best practices related to RR&T, including the development of standard operating procedures, supply chain management, and cross training. Given the consistent and compelling awareness of the importance of RR&T expressed by core directors in 2017 and 2021 contrasted with the lack of apparent improvements over this time period, the authors recommend an adoption of RR&T statements by all core laboratories. Adhering to the RR&T guidelines will result in more efficient training, better compliance, and improved experimental approaches empowering cores to become "rigor champions."
Collapse
Affiliation(s)
- Andrew W Ott
- Northwestern University EvanstonIllinois60208 USA
| | | | | | | | - Susan M Meyn
- Vanderbilt University Medical Center NashvilleTennessee37232 USA
| | - Sheenah M Mische
- New York University Langone Medical Center New YorkNew York10016 USA
| | - Douglas J Taatjes
- Larner College of Medicine University of Vermont BurlingtonVermont05405 USA
| | | | | |
Collapse
|
16
|
Curtis MJ, Alexander SPH, Cirino G, George CH, Kendall DA, Insel PA, Izzo AA, Ji Y, Panettieri RA, Patel HH, Sobey CG, Stanford SC, Stanley P, Stefanska B, Stephens GJ, Teixeira MM, Vergnolle N, Ahluwalia A. Planning experiments: Updated guidance on experimental design and analysis and their reporting III. Br J Pharmacol 2022; 179:3907-3913. [PMID: 35673806 DOI: 10.1111/bph.15868] [Citation(s) in RCA: 216] [Impact Index Per Article: 108.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022] Open
Abstract
Scientists who plan to publish in British Journal of Pharmacology (BJP) must read this article before undertaking a study. This editorial provides guidance for the design of experiments. We have published previously two guidance documents on experimental design and analysis (Curtis et al., 2015; Curtis et al., 2018). This update clarifies and simplifies the requirements on design and analysis for BJP manuscripts. This editorial also details updated requirements following an audit and discussion on best practice by the BJP editorial board. Explanations for the requirements are provided in the previous articles. Here, we address new issues that have arisen in the course of handling manuscripts and emphasise three aspects of design that continue to present the greatest challenge to authors: randomisation, blinded analysis and balance of group sizes.
Collapse
Affiliation(s)
| | | | | | | | | | - Paul A Insel
- University of California, San Diego, California, USA
| | | | - Yong Ji
- Nanjing Medical University, Nanjing, China
| | | | - Hemal H Patel
- University of California, San Diego, California, USA
| | | | | | | | | | | | | | | | - Amrita Ahluwalia
- William Harvey Research Institute, Queen Mary University of London, London, UK
| |
Collapse
|
17
|
Menke J, Eckmann P, Ozyurt IB, Roelandse M, Anderson N, Grethe J, Gamst A, Bandrowski A. Establishing Institutional Scores With the Rigor and Transparency Index: Large-scale Analysis of Scientific Reporting Quality. J Med Internet Res 2022; 24:e37324. [PMID: 35759334 PMCID: PMC9274430 DOI: 10.2196/37324] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2022] [Revised: 05/10/2022] [Accepted: 05/23/2022] [Indexed: 12/11/2022] Open
Abstract
BACKGROUND Improving rigor and transparency measures should lead to improvements in reproducibility across the scientific literature; however, the assessment of measures of transparency tends to be very difficult if performed manually. OBJECTIVE This study addresses the enhancement of the Rigor and Transparency Index (RTI, version 2.0), which attempts to automatically assess the rigor and transparency of journals, institutions, and countries using manuscripts scored on criteria found in reproducibility guidelines (eg, Materials Design, Analysis, and Reporting checklist criteria). METHODS The RTI tracks 27 entity types using natural language processing techniques such as Bidirectional Long Short-term Memory Conditional Random Field-based models and regular expressions; this allowed us to assess over 2 million papers accessed through PubMed Central. RESULTS Between 1997 and 2020 (where data were readily available in our data set), rigor and transparency measures showed general improvement (RTI 2.29 to 4.13), suggesting that authors are taking the need for improved reporting seriously. The top-scoring journals in 2020 were the Journal of Neurochemistry (6.23), British Journal of Pharmacology (6.07), and Nature Neuroscience (5.93). We extracted the institution and country of origin from the author affiliations to expand our analysis beyond journals. Among institutions publishing >1000 papers in 2020 (in the PubMed Central open access set), Capital Medical University (4.75), Yonsei University (4.58), and University of Copenhagen (4.53) were the top performers in terms of RTI. In country-level performance, we found that Ethiopia and Norway consistently topped the RTI charts of countries with 100 or more papers per year. In addition, we tested our assumption that the RTI may serve as a reliable proxy for scientific replicability (ie, a high RTI represents papers containing sufficient information for replication efforts). Using work by the Reproducibility Project: Cancer Biology, we determined that replication papers (RTI 7.61, SD 0.78) scored significantly higher (P<.001) than the original papers (RTI 3.39, SD 1.12), which according to the project required additional information from authors to begin replication efforts. CONCLUSIONS These results align with our view that RTI may serve as a reliable proxy for scientific replicability. Unfortunately, RTI measures for journals, institutions, and countries fall short of the replicated paper average. If we consider the RTI of these replication studies as a target for future manuscripts, more work will be needed to ensure that the average manuscript contains sufficient information for replication attempts.
Collapse
Affiliation(s)
- Joe Menke
- Center for Research in Biological Systems, University of California, San Diego, La Jolla, CA, United States
- SciCrunch Inc., San Diego, CA, United States
| | - Peter Eckmann
- SciCrunch Inc., San Diego, CA, United States
- Department of Neuroscience, University of California, San Diego, La Jolla, CA, United States
| | - Ibrahim Burak Ozyurt
- SciCrunch Inc., San Diego, CA, United States
- Department of Neuroscience, University of California, San Diego, La Jolla, CA, United States
| | | | | | - Jeffrey Grethe
- SciCrunch Inc., San Diego, CA, United States
- Department of Neuroscience, University of California, San Diego, La Jolla, CA, United States
| | - Anthony Gamst
- Department of Mathematics, University of California, San Diego, CA, United States
| | - Anita Bandrowski
- SciCrunch Inc., San Diego, CA, United States
- Department of Neuroscience, University of California, San Diego, La Jolla, CA, United States
| |
Collapse
|
18
|
Schulz R, Barnett A, Bernard R, Brown NJL, Byrne JA, Eckmann P, Gazda MA, Kilicoglu H, Prager EM, Salholz-Hillel M, Ter Riet G, Vines T, Vorland CJ, Zhuang H, Bandrowski A, Weissgerber TL. Is the future of peer review automated? BMC Res Notes 2022; 15:203. [PMID: 35690782 PMCID: PMC9188010 DOI: 10.1186/s13104-022-06080-6] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2022] [Accepted: 05/18/2022] [Indexed: 12/19/2022] Open
Abstract
The rising rate of preprints and publications, combined with persistent inadequate reporting practices and problems with study design and execution, have strained the traditional peer review system. Automated screening tools could potentially enhance peer review by helping authors, journal editors, and reviewers to identify beneficial practices and common problems in preprints or submitted manuscripts. Tools can screen many papers quickly, and may be particularly helpful in assessing compliance with journal policies and with straightforward items in reporting guidelines. However, existing tools cannot understand or interpret the paper in the context of the scientific literature. Tools cannot yet determine whether the methods used are suitable to answer the research question, or whether the data support the authors' conclusions. Editors and peer reviewers are essential for assessing journal fit and the overall quality of a paper, including the experimental design, the soundness of the study's conclusions, potential impact and innovation. Automated screening tools cannot replace peer review, but may aid authors, reviewers, and editors in improving scientific papers. Strategies for responsible use of automated tools in peer review may include setting performance criteria for tools, transparently reporting tool performance and use, and training users to interpret reports.
Collapse
Affiliation(s)
- Robert Schulz
- BIH QUEST Center for Responsible Research, Berlin Institute of Health at Charité Universitätsmedizin Berlin, Berlin, Germany
| | - Adrian Barnett
- Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health & Social Work, Queensland University of Technology, Brisbane, QLD, Australia
| | - René Bernard
- NeuroCure Cluster of Excellence, Charité Universitätsmedizin Berlin, Berlin, Germany
| | | | - Jennifer A Byrne
- Faculty of Medicine and Health, New South Wales Health Pathology, The University of Sydney, New South Wales, Australia
| | - Peter Eckmann
- Department of Neuroscience, University of California, San Diego, La Jolla, CA, USA
| | - Małgorzata A Gazda
- UMR 3525, Institut Pasteur, Université de Paris, CNRS, INSERM UA12, Comparative Functional Genomics group, Paris, France
| | - Halil Kilicoglu
- School of Information Sciences, University of Illinois Urbana-Champaign, Champaign, IL, USA
| | - Eric M Prager
- Translational Research and Development, Cohen Veterans Bioscience, New York, NY, USA
| | - Maia Salholz-Hillel
- BIH QUEST Center for Responsible Research, Berlin Institute of Health at Charité Universitätsmedizin Berlin, Berlin, Germany
| | - Gerben Ter Riet
- Faculty of Health, Center of Expertise Urban Vitality, Amsterdam University of Applied Science, Amsterdam, The Netherlands
| | - Timothy Vines
- DataSeer Research Data Services Ltd, Vancouver, BC, Canada
| | - Colby J Vorland
- Indiana University School of Public Health-Bloomington, Bloomington, IN, USA
| | - Han Zhuang
- School of Information Studies, Syracuse University, Syracuse, NY, USA
| | - Anita Bandrowski
- Department of Neuroscience, University of California, San Diego, La Jolla, CA, USA
| | - Tracey L Weissgerber
- BIH QUEST Center for Responsible Research, Berlin Institute of Health at Charité Universitätsmedizin Berlin, Berlin, Germany.
| |
Collapse
|
19
|
Sander T, Ghanawi J, Wilson E, Muhammad S, Macleod M, Kahlert UD. Meta-analysis on reporting practices as a source of heterogeneity in in vitro cancer research. BMJ OPEN SCIENCE 2022; 6:e100272. [PMID: 35721833 PMCID: PMC9171230 DOI: 10.1136/bmjos-2021-100272] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2021] [Accepted: 05/09/2022] [Indexed: 11/13/2022] Open
Abstract
Objectives Heterogeneity of results of exact same research experiments oppose a significant socioeconomic burden. Insufficient methodological reporting is likely to be one of the contributors to results heterogeneity; however, little knowledge on reporting habits of in vitro cancer research and their effects on results reproducibility is available. Exemplified by a commonly performed in vitro assay, we aim to fill this knowledge gap and to derive recommendations necessary for reproducible, robust and translational preclinical science. Methods Here, we use systematic review to describe reporting practices in in vitro glioblastoma research using the Uppsala-87 Malignant Glioma (U-87 MG) cell line and perform multilevel random-effects meta-analysis followed by meta-regression to explore sources of heterogeneity within that literature, and any associations between reporting characteristics and reported findings. Literature that includes experiments measuring the effect of temozolomide on the viability of U-87 MG cells is searched on three databases (Embase, PubMed and Web of Science). Results In 137 identified articles, the methodological reporting is incomplete, for example, medium glucose level and cell density are reported in only 21.2% and 16.8% of the articles. After adjustments for different drug concentrations and treatment durations, the results heterogeneity across the studies (I2=68.5%) is concerningly large. Differences in culture medium glucose level are a driver of this heterogeneity. However, infrequent reporting of most experimental parameters limits the analysis of reproducibility moderating parameters. Conclusions Our results further support the ongoing efforts of establishing consensus reporting practices to elevate durability of results. By doing so, this work can raise awareness of how stricter reporting may help to improve the frequency of successful translation of preclinical results into human application. The authors received no specific funding for this work. A preregistered protocol is available at the Open Science Framework (https://osf.io/9k3dq).
Collapse
Affiliation(s)
- Timo Sander
- Department of Neurosurgery, Medical Faculty, Heinrich Heine University, Düsseldorf, Germany
| | | | - Emma Wilson
- Centre for Clinical Brain Sciences, The University of Edinburgh Medical School, Edinburgh, UK
| | - Sajjad Muhammad
- Department of Neurosurgery, Medical Faculty, Heinrich Heine University, Düsseldorf, Germany
| | - Malcolm Macleod
- Centre for Clinical Brain Sciences, The University of Edinburgh Medical School, Edinburgh, UK
| | - Ulf Dietrich Kahlert
- Department of Molecular and Experimental Surgery, Clinic for General, Visceral, Vascular and Transplant Surgery, Otto von Guericke Universität Magdeburg, Magdeburg, Germany
| |
Collapse
|
20
|
Dirnagl U, Duda GN, Grainger DW, Reinke P, Roubenoff R. Reproducibility, relevance and reliability as barriers to efficient and credible biomedical technology translation. Adv Drug Deliv Rev 2022; 182:114118. [PMID: 35066104 DOI: 10.1016/j.addr.2022.114118] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2021] [Revised: 01/14/2022] [Accepted: 01/15/2022] [Indexed: 12/23/2022]
Abstract
Biomedical research accuracy and relevance for improving healthcare are increasingly identified as costly problems. Basic research data quality, reporting and methodology, and reproducibility are common factors implicated in this challenge. Preclinical models of disease and therapy, largely conducted in rodents, have known deficiencies in replicating most human conditions. Their translation to human results is acknowledged to be poor for decades. Clinical data quality and quantity is also recognized as deficient; gold standard randomized clinical trials are expensive. Few solid conclusions from clinical studies are replicable and many remain unpublished. The translational pathway from fundamental biomedical research through to innovative solutions handed to clinical practitioners is therefore highly inefficient and costly in terms of wasted resources, early claims from fundamental discoveries never witnessed in humans, and few new, improved solutions available clinically for myriad diseases. Improving this biomedical research strategy and resourcing for reliability, translational relevance, reproducibility and clinical impact requires careful analysis and consistent enforcement at both funding and peer review levels.
Collapse
Affiliation(s)
- Ulrich Dirnagl
- Department of Experimental Neurology, Charité - Universitätsmedizin Berlin, Germany; QUEST Center for Responsible Research, Berlin Institute of Health, Germany
| | - Georg N Duda
- Berlin Institute of Health (BIH) Center for Regenerative Therapies (BCRT), Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Germany; Julius Wolff Institute for Biomechanics and Musculoskeletal Regeneration, Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Germany
| | - David W Grainger
- Department of Pharmaceutics and Pharmaceutical Chemistry, Health Sciences, University of Utah, Salt Lake City, UT 84112 USA; Department of Biomedical Engineering, University of Utah, Salt Lake City, UT 84112 USA.
| | - Petra Reinke
- Berlin Institute of Health (BIH) Center for Regenerative Therapies (BCRT), Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Germany; Berlin Center for Advanced Therapies (BeCAT), Charité - Universitaetsmedizin Berlin, 13353 Berlin, Germany
| | - Ronenn Roubenoff
- Novartis Institutes for Biomedical Research, Cambridge, Basel, Massachusetts, Switzerland
| |
Collapse
|
21
|
Hepkema WM, Horbach SPJM, Hoek JM, Halffman W. Misidentified biomedical resources: Journal guidelines are not a quick fix. Int J Cancer 2021; 150:1233-1243. [PMID: 34807460 PMCID: PMC9300184 DOI: 10.1002/ijc.33882] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2021] [Revised: 10/14/2021] [Accepted: 11/02/2021] [Indexed: 01/22/2023]
Abstract
Biomedical researchers routinely use a variety of biological models and resources, such as cultured cell lines, antibodies and laboratory animals. Unfortunately, these resources are not flawless: cell lines can be misidentified; for antibodies, problems with specificity, lot‐to‐lot consistency and sensitivity are common; and the reliability of animal models is questioned due to poor translation of animal studies to human clinical trials. In some cases, these problems can render the results of a study meaningless. As a response, some journals have implemented guidelines regarding the use and reporting of cell lines, antibodies and laboratory animals. In our study we use a portfolio of existing and newly created datasets to investigate identification and authentication information of cell lines, antibodies and organisms before and after guideline introduction, compared to journals without guidelines. We observed a general improvement of reporting quality over time, which the implementation of guidelines accelerated only in some cases. We therefore conclude that the effectiveness of journal guidelines is likely to be context dependent, affected by factors such as implementation conditions, research community support and monitoring and resource availability. Hence, journal reporting guidelines in themselves are not a quick fix to repair shortcomings in biomedical resource documentation, even though they can be part of the solution.
Collapse
Affiliation(s)
- Wytske M Hepkema
- Institute of Sociology, Technische Universität Berlin, Berlin, Germany
| | - Serge P J M Horbach
- Danish Centre for Studies in Research and Research Policy, Aarhus University, Aarhus, Denmark.,Centre for Science and Technology Studies, Leiden University, Leiden, The Netherlands
| | - Joyce M Hoek
- Department of Psychology, University of Groningen, Groningen, The Netherlands
| | - Willem Halffman
- Institute for Science in Society, Radboud University Nijmegen, Nijmegen, The Netherlands
| |
Collapse
|
22
|
Wang Q, Liao J, Lapata M, Macleod M. Risk of bias assessment in preclinical literature using natural language processing. Res Synth Methods 2021; 13:368-380. [PMID: 34709718 PMCID: PMC9298308 DOI: 10.1002/jrsm.1533] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2021] [Revised: 10/01/2021] [Accepted: 10/09/2021] [Indexed: 12/09/2022]
Abstract
We sought to apply natural language processing to the task of automatic risk of bias assessment in preclinical literature, which could speed the process of systematic review, provide information to guide research improvement activity, and support translation from preclinical to clinical research. We use 7840 full‐text publications describing animal experiments with yes/no annotations for five risk of bias items. We implement a series of models including baselines (support vector machine, logistic regression, random forest), neural models (convolutional neural network, recurrent neural network with attention, hierarchical neural network) and models using BERT with two strategies (document chunk pooling and sentence extraction). We tune hyperparameters to obtain the highest F1 scores for each risk of bias item on the validation set and compare evaluation results on the test set to our previous regular expression approach. The F1 scores of best models on test set are 82.0% for random allocation, 81.6% for blinded assessment of outcome, 82.6% for conflict of interests, 91.4% for compliance with animal welfare regulations and 46.6% for reporting animals excluded from analysis. Our models significantly outperform regular expressions for four risk of bias items. For random allocation, blinded assessment of outcome, conflict of interests and animal exclusions, neural models achieve good performance; for animal welfare regulations, BERT model with a sentence extraction strategy works better. Convolutional neural networks are the overall best models. The tool is publicly available which may contribute to the future monitoring of risk of bias reporting for research improvement activities.
Collapse
Affiliation(s)
- Qianying Wang
- Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, UK
| | - Jing Liao
- Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, UK
| | - Mirella Lapata
- School of Informatics, University of Edinburgh, Edinburgh, UK
| | - Malcolm Macleod
- Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, UK
| |
Collapse
|
23
|
Bannach-Brown A, Hair K, Bahor Z, Soliman N, Macleod M, Liao J. Technological advances in preclinical meta-research. BMJ OPEN SCIENCE 2021; 5:e100131. [PMID: 35047701 PMCID: PMC8647618 DOI: 10.1136/bmjos-2020-100131] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/16/2023] Open
Affiliation(s)
- Alexandra Bannach-Brown
- Berlin Institute of Health, QUEST Center, Charité Universitätsmedizin Berlin, Berlin, Germany
- Institute for Evidence-Based Practice, Bond University, Robina, Queensland, Australia
| | - Kaitlyn Hair
- Centre for Clinical Brain Sciences, The University of Edinburgh Edinburgh Medical School, Edinburgh, Scotland, UK
| | - Zsanett Bahor
- Centre for Clinical Brain Sciences, The University of Edinburgh Edinburgh Medical School, Edinburgh, Scotland, UK
| | - Nadia Soliman
- Pain Research; Faculty of Medicine, Department of Surgery and Cancer, Imperial College London, London, Greater London, UK
| | - Malcolm Macleod
- Centre for Clinical Brain Sciences, The University of Edinburgh Edinburgh Medical School, Edinburgh, Scotland, UK
| | - Jing Liao
- Centre for Clinical Brain Sciences, The University of Edinburgh Edinburgh Medical School, Edinburgh, Scotland, UK
| |
Collapse
|
24
|
Akhlaghi M, Infante-Sainz R, Roukema BF, Khellat M, Valls-Gabaud D, Baena-Galle R, A. Barba L, Gesing S. Toward Long-Term and Archivable Reproducibility. Comput Sci Eng 2021. [DOI: 10.1109/mcse.2021.3072860] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
25
|
Macleod M, Collings AM, Graf C, Kiermer V, Mellor D, Swaminathan S, Sweet D, Vinson V. The MDAR (Materials Design Analysis Reporting) Framework for transparent reporting in the life sciences. Proc Natl Acad Sci U S A 2021; 118:e2103238118. [PMID: 33893240 PMCID: PMC8092464 DOI: 10.1073/pnas.2103238118] [Citation(s) in RCA: 33] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/17/2022] Open
Affiliation(s)
- Malcolm Macleod
- Edinburgh CAMARADES group, Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh EH8 9YL, United Kingdom;
| | | | - Chris Graf
- John Wiley & Sons, Oxford OX4 2DQ, United Kingdom
| | | | - David Mellor
- Center for Open Science, Charlottesville, VA 22903;
| | | | | | - Valda Vinson
- Science, American Association for the Advancement of Science, Washington, DC 20005
| |
Collapse
|
26
|
Howat AM, Mulhern A, Logan HF, Redvers-Mutton G, Routledge C, Clark J. Converting Access Microbiology to an open research platform: focus group and AI review tool research results. Access Microbiol 2021; 3:000232. [PMID: 34151179 PMCID: PMC8208759 DOI: 10.1099/acmi.0.000232] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2021] [Indexed: 11/18/2022] Open
Abstract
The Microbiology Society will be launching an open research platform in October 2021. Developed using funding from the Wellcome Trust and the Howard Hughes Medical Institute (HHMI), the platform will combine our current sound-science journal, Access Microbiology, with artificial intelligence (AI) review tools and many of the elements of a preprint server. In an effort to improve the rigour, reproducibility and transparency of the academic record, the Access Microbiology platform will host both preprints of articles and their Version of Record (VOR) publications, as well as the reviewer reports, Editor's decision, authors' response to reviewers and the AI review reports. To ensure the platform meets the needs of our community, in February 2020 we conducted focus group meetings with various stakeholders. Using articles previously submitted to Access Microbiology, we undertook testing of a range of potential AI review tools and investigated the technical feasibility and utility of including these tools as part of the platform. In keeping with the open and transparent ethos of the platform, we present here a summary of the focus group feedback and AI review tool testing.
Collapse
|
27
|
Kilicoglu H, Rosemblat G, Hoang L, Wadhwa S, Peng Z, Malički M, Schneider J, Ter Riet G. Toward assessing clinical trial publications for reporting transparency. J Biomed Inform 2021; 116:103717. [PMID: 33647518 PMCID: PMC8112250 DOI: 10.1016/j.jbi.2021.103717] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2020] [Revised: 02/14/2021] [Accepted: 02/15/2021] [Indexed: 10/22/2022]
Abstract
OBJECTIVE To annotate a corpus of randomized controlled trial (RCT) publications with the checklist items of CONSORT reporting guidelines and using the corpus to develop text mining methods for RCT appraisal. METHODS We annotated a corpus of 50 RCT articles at the sentence level using 37 fine-grained CONSORT checklist items. A subset (31 articles) was double-annotated and adjudicated, while 19 were annotated by a single annotator and reconciled by another. We calculated inter-annotator agreement at the article and section level using MASI (Measuring Agreement on Set-Valued Items) and at the CONSORT item level using Krippendorff's α. We experimented with two rule-based methods (phrase-based and section header-based) and two supervised learning approaches (support vector machine and BioBERT-based neural network classifiers), for recognizing 17 methodology-related items in the RCT Methods sections. RESULTS We created CONSORT-TM consisting of 10,709 sentences, 4,845 (45%) of which were annotated with 5,246 labels. A median of 28 CONSORT items (out of possible 37) were annotated per article. Agreement was moderate at the article and section levels (average MASI: 0.60 and 0.64, respectively). Agreement varied considerably among individual checklist items (Krippendorff's α= 0.06-0.96). The model based on BioBERT performed best overall for recognizing methodology-related items (micro-precision: 0.82, micro-recall: 0.63, micro-F1: 0.71). Combining models using majority vote and label aggregation further improved precision and recall, respectively. CONCLUSION Our annotated corpus, CONSORT-TM, contains more fine-grained information than earlier RCT corpora. Low frequency of some CONSORT items made it difficult to train effective text mining models to recognize them. For the items commonly reported, CONSORT-TM can serve as a testbed for text mining methods that assess RCT transparency, rigor, and reliability, and support methods for peer review and authoring assistance. Minor modifications to the annotation scheme and a larger corpus could facilitate improved text mining models. CONSORT-TM is publicly available at https://github.com/kilicogluh/CONSORT-TM.
Collapse
Affiliation(s)
- Halil Kilicoglu
- School of Information Sciences, University of Illinois at Urbana-Champaign, Champaign, IL, USA; U.S. National Library of Medicine, National Institutes of Health, Bethesda, MD, USA.
| | - Graciela Rosemblat
- U.S. National Library of Medicine, National Institutes of Health, Bethesda, MD, USA
| | - Linh Hoang
- School of Information Sciences, University of Illinois at Urbana-Champaign, Champaign, IL, USA
| | - Sahil Wadhwa
- Department of Statistics, University of Illinois at Urbana-Champaign, Champaign, IL, USA
| | - Zeshan Peng
- U.S. National Library of Medicine, National Institutes of Health, Bethesda, MD, USA
| | - Mario Malički
- Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA
| | - Jodi Schneider
- School of Information Sciences, University of Illinois at Urbana-Champaign, Champaign, IL, USA
| | - Gerben Ter Riet
- Urban Vitality Center of Expertise, Faculty of Health, Amsterdam University of Applied Sciences, Amsterdam, the Netherlands; Department of Cardiology Heart Center, Amsterdam UMC, University of Amsterdam, the Netherlands
| |
Collapse
|
28
|
Nüst D, Eglen SJ. CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility. F1000Res 2021; 10:253. [PMID: 34367614 PMCID: PMC8311796 DOI: 10.12688/f1000research.51738.2] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 06/15/2021] [Indexed: 11/20/2022] Open
Abstract
The traditional scientific paper falls short of effectively communicating computational research. To help improve this situation, we propose a system by which the computational workflows underlying research articles are checked. The CODECHECK system uses open infrastructure and tools and can be integrated into review and publication processes in multiple ways. We describe these integrations along multiple dimensions (importance, who, openness, when). In collaboration with academic publishers and conferences, we demonstrate CODECHECK with 25 reproductions of diverse scientific publications. These CODECHECKs show that asking for reproducible workflows during a collaborative review can effectively improve executability. While CODECHECK has clear limitations, it may represent a building block in Open Science and publishing ecosystems for improving the reproducibility, appreciation, and, potentially, the quality of non-textual research artefacts. The CODECHECK website can be accessed here: https://codecheck.org.uk/.
Collapse
Affiliation(s)
- Daniel Nüst
- Institute for Geoinformatics, University of Münster, Münster, Germany
| | - Stephen J. Eglen
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK
| |
Collapse
|
29
|
Nüst D, Eglen SJ. CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility. F1000Res 2021; 10:253. [PMID: 34367614 PMCID: PMC8311796 DOI: 10.12688/f1000research.51738.1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 03/22/2021] [Indexed: 11/08/2023] Open
Abstract
The traditional scientific paper falls short of effectively communicating computational research. To help improve this situation, we propose a system by which the computational workflows underlying research articles are checked. The CODECHECK system uses open infrastructure and tools and can be integrated into review and publication processes in multiple ways. We describe these integrations along multiple dimensions (importance, who, openness, when). In collaboration with academic publishers and conferences, we demonstrate CODECHECK with 25 reproductions of diverse scientific publications. These CODECHECKs show that asking for reproducible workflows during a collaborative review can effectively improve executability. While CODECHECK has clear limitations, it may represent a building block in Open Science and publishing ecosystems for improving the reproducibility, appreciation, and, potentially, the quality of non-textual research artefacts. The CODECHECK website can be accessed here: https://codecheck.org.uk/.
Collapse
Affiliation(s)
- Daniel Nüst
- Institute for Geoinformatics, University of Münster, Münster, Germany
| | - Stephen J. Eglen
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK
| |
Collapse
|
30
|
Serghiou S, Contopoulos-Ioannidis DG, Boyack KW, Riedel N, Wallach JD, Ioannidis JPA. Assessment of transparency indicators across the biomedical literature: How open is open? PLoS Biol 2021; 19:e3001107. [PMID: 33647013 PMCID: PMC7951980 DOI: 10.1371/journal.pbio.3001107] [Citation(s) in RCA: 60] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2020] [Revised: 03/11/2021] [Accepted: 01/19/2021] [Indexed: 12/16/2022] Open
Abstract
Recent concerns about the reproducibility of science have led to several calls for more open and transparent research practices and for the monitoring of potential improvements over time. However, with tens of thousands of new biomedical articles published per week, manually mapping and monitoring changes in transparency is unrealistic. We present an open-source, automated approach to identify 5 indicators of transparency (data sharing, code sharing, conflicts of interest disclosures, funding disclosures, and protocol registration) and apply it across the entire open access biomedical literature of 2.75 million articles on PubMed Central (PMC). Our results indicate remarkable improvements in some (e.g., conflict of interest [COI] disclosures and funding disclosures), but not other (e.g., protocol registration and code sharing) areas of transparency over time, and map transparency across fields of science, countries, journals, and publishers. This work has enabled the creation of a large, integrated, and openly available database to expedite further efforts to monitor, understand, and promote transparency and reproducibility in science.
Collapse
Affiliation(s)
- Stylianos Serghiou
- Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, California, United States of America
- Meta-Research Innovation Center at Stanford (METRICS), Stanford School of Medicine, Stanford, California, United States of America
| | - Despina G. Contopoulos-Ioannidis
- Division of Infectious Diseases, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States of America
| | - Kevin W. Boyack
- SciTech Strategies, Inc., Albuquerque, New Mexico, United States of America
| | - Nico Riedel
- Berlin Institute of Health, QUEST Center for Transforming Biomedical Research, Berlin, Germany
| | - Joshua D. Wallach
- Department of Environmental Health Sciences, Yale School of Public Health, New Haven, Connecticut, United States of America
| | - John P. A. Ioannidis
- Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, California, United States of America
- Meta-Research Innovation Center at Stanford (METRICS), Stanford School of Medicine, Stanford, California, United States of America
- Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, California, United States of America
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California, United States of America
- Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, California, United States of America
| |
Collapse
|
31
|
Weissgerber T, Riedel N, Kilicoglu H, Labbé C, Eckmann P, Ter Riet G, Byrne J, Cabanac G, Capes-Davis A, Favier B, Saladi S, Grabitz P, Bannach-Brown A, Schulz R, McCann S, Bernard R, Bandrowski A. Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility? Nat Med 2021; 27:6-7. [PMID: 33432174 PMCID: PMC8177099 DOI: 10.1038/s41591-020-01203-7] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
- Tracey Weissgerber
- Quality | Ethics | Open Science | Translation (QUEST), Berlin Institute of Health, Berlin, Germany.
- Charité-Universitätsmedizin Berlin, Berlin, Germany.
| | - Nico Riedel
- Quality | Ethics | Open Science | Translation (QUEST), Berlin Institute of Health, Berlin, Germany
| | - Halil Kilicoglu
- School of Information Sciences, University of Illinois at Urbana-Champaign, Champaign, IL, USA
| | - Cyril Labbé
- University Grenoble Alpes, CNRS, Grenoble INP, LIG, Grenoble, France
| | - Peter Eckmann
- Department of Neuroscience, University of California, San Diego, La Jolla, CA, USA
- SciCrunch Inc., San Diego, CA, USA
| | - Gerben Ter Riet
- Department of Cardiology, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
- Urban Vitality Center of Expertise, Amsterdam University of Applied Sciences, Amsterdam, the Netherlands
| | - Jennifer Byrne
- New South Wales Health Statewide Biobank, New South Wales Health Pathology, Sydney, New South Wales, Australia
- Faculty of Medicine and Health, University of Sydney, Sydney, New South Wales, Australia
| | | | - Amanda Capes-Davis
- CellBank Australia, Children's Medical Research Institute and The University of Sydney, Westmead, NSW, Australia
| | | | - Shyam Saladi
- California Institute of Technology, Pasadena, CA, USA
| | - Peter Grabitz
- Quality | Ethics | Open Science | Translation (QUEST), Berlin Institute of Health, Berlin, Germany
- Charité-Universitätsmedizin Berlin, Berlin, Germany
| | - Alexandra Bannach-Brown
- Quality | Ethics | Open Science | Translation (QUEST), Berlin Institute of Health, Berlin, Germany
| | - Robert Schulz
- Quality | Ethics | Open Science | Translation (QUEST), Berlin Institute of Health, Berlin, Germany
- Charité-Universitätsmedizin Berlin, Berlin, Germany
| | - Sarah McCann
- Quality | Ethics | Open Science | Translation (QUEST), Berlin Institute of Health, Berlin, Germany
- Charité-Universitätsmedizin Berlin, Berlin, Germany
| | - Rene Bernard
- NeuroCure Cluster of Excellence, Charité-Universitätsmedizin Berlin, corporate member of the Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany
| | - Anita Bandrowski
- Department of Neuroscience, University of California, San Diego, La Jolla, CA, USA
- SciCrunch Inc., San Diego, CA, USA
| |
Collapse
|
32
|
Alves CPDL, Costa GGD. Transparência e integridade em pesquisa: dos problemas às potenciais soluções. REVISTA BRASILEIRA DE GERIATRIA E GERONTOLOGIA 2021. [DOI: 10.1590/1981-22562021024.210239] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022] Open
|