Independent discussion sections for improving inferential reproducibility in published research

Automatic categorization of self-acknowledged limitations in randomized controlled trial publications

OBJECTIVE

Acknowledging study limitations in a scientific publication is a crucial element in scientific transparency and progress. However, limitation reporting is often inadequate. Natural language processing (NLP) methods could support automated reporting checks, improving research transparency. In this study, our objective was to develop a dataset and NLP methods to detect and categorize self-acknowledged limitations (e.g., sample size, blinding) reported in randomized controlled trial (RCT) publications.

METHODS

We created a data model of limitation types in RCT studies and annotated a corpus of 200 full-text RCT publications using this data model. We fine-tuned BERT-based sentence classification models to recognize the limitation sentences and their types. To address the small size of the annotated corpus, we experimented with data augmentation approaches, including Easy Data Augmentation (EDA) and Prompt-Based Data Augmentation (PromDA). We applied the best-performing model to a set of about 12K RCT publications to characterize self-acknowledged limitations at larger scale.

RESULTS

Our data model consists of 15 categories and 24 sub-categories (e.g., Population and its sub-category DiagnosticCriteria). We annotated 1090 instances of limitation types in 952 sentences (4.8 limitation sentences and 5.5 limitation types per article). A fine-tuned PubMedBERT model for limitation sentence classification improved upon our earlier model by about 1.5 absolute percentage points in F1 score (0.821 vs. 0.8) with statistical significance (p<.001). Our best-performing limitation type classification model, PubMedBERT fine-tuning with PromDA (Output View), achieved an F1 score of 0.7, improving upon the vanilla PubMedBERT model by 2.7 percentage points, with statistical significance (p<.001).

CONCLUSION

The model could support automated screening tools which can be used by journals to draw the authors' attention to reporting issues. Automatic extraction of limitations from RCT publications could benefit peer review and evidence synthesis, and support advanced methods to search and aggregate the evidence from the clinical trial literature.

Trials and tribulations of statistical significance in biochemistry and omics

Over recent years many statisticians and researchers have highlighted that statistical inference would benefit from a better use and understanding of hypothesis testing, p-values, and statistical significance. We highlight three recommendations in the context of biochemical sciences. First recommendation: to improve the biological interpretation of biochemical data, do not use p-values (or similar test statistics) as thresholded values to select biomolecules. Second recommendation: to improve comparison among studies and to achieve robust knowledge, perform complete reporting of data. Third recommendation: statistical analyses should be reported completely with exact numbers (not as asterisks or inequalities). Owing to the high number of variables, a better use of statistics is of special importance in omic studies.

Spin within systematic review abstracts on antiplatelet therapies after acute coronary syndrome: a cross-sectional study

OBJECTIVES

Spin is a reporting practice in which study results are misrepresented by overestimating efficacy or underestimating harm. Prevalence of spin varies between clinical specialties, and estimates are based almost entirely on clinical trials. Little is known about spin in systematic reviews.

DESIGN

We performed a cross-sectional analysis searching MEDLINE and Embase for systematic reviews and meta-analyses pertaining to antiplatelet therapies following acute coronary syndrome on 2 June 2020. Data were extracted evaluating the presence of spin and study characteristics, including methodological quality as rated by A MeaSurement Tool to Assess systematic Reviews (AMSTAR-2). All data extraction was conducted in a masked, duplicate manner from 2 June 2020 to 26 June 2020.

PARTICIPANTS AND SETTING

Not applicable.

PRIMARY AND SECONDARY OUTCOME MEASURES

We assessed abstracts of systematic reviews on antiplatelet therapy following acute coronary syndrome and evaluated the prevalence of the nine most severe types of spin. We additionally explored associations between spin and certain study characteristics, including quality.

RESULTS

Our searches returned 15 263 articles, and 185 systematic reviews met inclusion criteria. Of these 185 reviews, 31.9% (59/185) contained some form of spin in the abstract. Seven forms of spin (1, 2, 3, 4, 5, 7 and 9) among the nine most severe were identified. No instances of types 6 or 8 were found. There were no statistically significant relationships between spin and the evaluated study characteristics or AMSTAR-2 appraisals.

CONCLUSIONS

Spin was present in abstracts for systematic reviews and meta-analyses; subsequent studies are needed to identify correlations between spin and specific study characteristics. There were no statistically significant associations between spin and study characteristics or AMSTAR-2 ratings; however, implementing changes will ensure that spin is reduced in the field of cardiology as well as other fields of medicine.

Fraud and retraction in perioperative medicine publications: what we learned and what can be implemented to prevent future recurrence

Fraud in medical publications is an increasing concern. In particular, disciplines related to perioperative medicine-including anaesthesia and critical care-currently hold the highest rankings in terms of retracted papers for research misconduct. The dominance of this dubious achievement is attributable to a limited number of researchers who have repeatedly committed scientific fraud. In the last three decades, six researchers have authored 421 of the 475 papers retracted in perioperative medicine. This narrative review reports on six cases of fabricated publication in perioperative medicine that resulted in the paper's retraction. The process that led to the unveiling of the fraud, the impact on clinical practice, and changes in regulatory mechanisms of scientific companies and governmental agencies' policies are also presented. Fraud in medical publications is a growing concern that affects perioperative medicine requiring a substantial number of papers to be retracted. The continuous control elicited by readers, by local institutional review boards, scientific journal reviewers, scientific societies and government agencies can play an important role in preserving the 'pact of trust' between authors, professionals and ultimately the relationship between doctors and patients.

Asymptomatic neurocognitive impairment is a risk for symptomatic decline over a 3-year study period

OBJECTIVE

To examine whether persons with asymptomatic neurocognitive impairment (ANI) were more likely to show progression to mild neurocognitive disorder or HIV-associated dementia than those who were neuropsychologically normal (NP-N).

DESIGN

Longitudinal observational cohort study.

METHODS

Study sample included 720 HIV-1 seropositive persons (317 with ANI and 403 NP-N) receiving care in Toronto, Canada [83% were on antiretroviral treatment; 71% had undetectable (<50 copies/ml) plasma HIVRNA]. Neuropsychological assessments were conducted at 12 months intervals for a median follow-up time of 34 months. Neuropsychological data were corrected for age, education, sex, and race/ethnicity, and corrected for practice effect at follow-ups. Progression to mild neurocognitive disorder and HIV-associated dementia at each time point was determined using the Global Deficit Score and presence of cognitive symptoms.

RESULTS

Over the follow-up period, 170 individuals (24%) progressed to symptomatic HIV-associated neurocognitive disorders (HAND). Persons with ANI were more likely to progress to symptomatic HAND than persons with NP-N after adjusting for baseline and time-varying confounders (adjusted hazards ratio: 1.88; 95% confidence interval: 1.37-2.60; P < 0.001). Female sex, depression, and cigarette smoking were associated with higher risk of progression to symptomatic HAND, but traditional HIV markers and antiretroviral treatment were not.

CONCLUSION

ANI is associated with a two-fold increased risk of progression to symptomatic HAND in a cohort with universal healthcare access. This represents the largest replication of comparable US results. Reproducibility of these findings indicate that routine monitoring of persons with ANI and exploration of clinical interventions to prevent or delay progression to symptomatic HAND are imperative.

SEARCH TERMS

HIV, HAND, HIV-associated dementia, cohort study, replicability, reproducibility.

Sample size, study length, and inadequate controls were the most common self-acknowledged limitations in manual therapy trials: A methodological review

OBJECTIVES

The aim of this study was to quantify and analyze the presence and type of self-acknowledged limitations (SALs) in a sample of manual therapy (MT) randomized controlled trials.

STUDY DESIGN AND SETTING

We randomly selected 120 MT trials. We extracted data related to SALs from the original reports and classified them into 12 categories. After data extraction, specific limitations within each category were identified. A descriptive analysis was performed using frequencies and percentages for qualitative variables.

RESULTS

The number of SALs per trial article ranged from 0 to 8, and more than two-thirds of trials acknowledged at least two different limitations. Despite its small proportion, 9% of trials did not report SALs. The most common limitation declared, in almost half of our sample, related to sample size (47.5%) followed by limitations related to study length and follow-up (33.3%) and inadequate controls (32.5%).

CONCLUSION

Our results indicate that at least two different limitations are consistently acknowledged in MT trial reports, the most common being those related to sample size, study length, follow-up, and inadequate controls. Analysis of the reasons behind the SALs gives some insights about the main difficulties in conducting research in this field and may help develop strategies to improve future research.