Application of Benford’s law: a valuable tool for detecting scientific papers with fabricated data?: A case study using proven falsified articles against a comparison group

Statistical techniques to assess publication integrity in groups of randomized trials: a narrative review

OBJECTIVES

To describe statistical tools available for assessing publication integrity of groups of randomized controlled trials (RCTs).

STUDY DESIGN AND SETTING

Narrative review.

RESULTS

Freely available statistical tools have been developed that compare the observed distributions of baseline variables with the expected distributions that would occur if successful randomization occurred. For continuous variables, the tools assess baseline means, baseline P values, and the occurrence of identical means and/or standard deviation. For categorical variables, they assess baseline P values, frequency counts for individual or all variables, numbers of trial participants randomized or withdrawing, and compare reported with independently calculated P values. The tools have been used to identify publication integrity concerns in RCTs from individual groups, and performed at an acceptable level in discriminating intentionally fabricated baseline summary data from genuine RCTs. The tools can be used when concerns have been raised about RCT(s) from an individual/group and when the whole body of their work is being examined, when conducting systematic reviews, and could be adapted to aid screening of RCTs at journal submission.

CONCLUSION

Statistical tools are useful for the assessment of publication integrity of groups of RCTs.

Investigating and preventing scientific misconduct using Benford's Law

Integrity and trust in that integrity are fundamental to academic research. However, procedures for monitoring the trustworthiness of research, and for investigating cases where concern about possible data fraud have been raised are not well established. Here we suggest a practical approach for the investigation of work suspected of fraudulent data manipulation using Benford's Law. This should be of value to both individual peer-reviewers and academic institutions and journals. In this, we draw inspiration from well-established practices of financial auditing. We provide synthesis of the literature on tests of adherence to Benford's Law, culminating in advice of a single initial test for digits in each position of numerical strings within a dataset. We also recommend further tests which may prove useful in the event that specific hypotheses regarding the nature of data manipulation can be justified. Importantly, our advice differs from the most common current implementations of tests of Benford's Law. Furthermore, we apply the approach to previously-published data, highlighting the efficacy of these tests in detecting known irregularities. Finally, we discuss the results of these tests, with reference to their strengths and limitations.

Flawed, futile, and fabricated-features that limit confidence in clinical research in pain and anaesthesia: a narrative review

The randomised controlled trial is the foundation of clinical research; yet there is concern that many trials have flaws in design, conduct, and reporting that undermine trustworthiness. Common flaws in trials include high risk of bias, small size, outcomes irrelevant to clinical care and patient's experience, and inability to detect efficacy even if present. These flaws carry forward into systematic reviews, which can confer the label of 'high-quality evidence' on inadequate data. Trials can be futile because their flaws mean that they cannot deliver any meaningful result in that different results in a small number of patients would be sufficient to change conclusions. Some trials have been discovered to be fabricated, the number of which is growing. The fields of anaesthesia and pain have more fabricated trials than other clinical fields, possibly because of increased vigilance. This narrative review examines these themes in depth whilst acknowledging an inescapable conclusion: that much of our clinical evidence is in trouble, and special measures are needed to bolster quality and confidence.

Assessing Research Misconduct in Randomized Controlled Trials

Randomized controlled trials (RCTs) serve as the pillar of evidence-based medicine and guide medical practice. Compromised data integrity in RCTs undermines the authority of this valuable tool for science and puts patients at risk. Although a large number of retractions due to data issues in obstetrics and gynecology have occurred in the past few years, many problematic RCTs could still go uncovered because in general there is insufficient willingness to envisage and confront research misconduct. In this article, we discuss the necessity of assessing research misconduct, summarize methods that have been applied in detecting previous cases of misconduct, and propose potential solutions. There is no established mechanism to monitor feedback on published articles and the current system that handles potential research misconduct is unsatisfactory. Fortunately, there are methods to assess data integrity in RCTs both with and without individual participant data. Investigations into research misconduct can be facilitated by assessing all publications from a leading author or author group to identify duplication and patterns of ongoing misconduct. There is a pressing need to improve the mechanism that investigates data manipulation. The mechanism that handles misconduct should prioritize the interests of patients and readers rather than trial authors and their institutions. An equally urgent issue is to establish mechanisms that prevent compromised trials from polluting evidence synthesis or misguiding practice.

Methods to assess research misconduct in health-related research: A scoping review

OBJECTIVE

To give an overview of the available methods to investigate research misconduct in health-related research.

STUDY DESIGN AND SETTING

In this scoping review, we conducted a literature search in MEDLINE, Embase, The Cochrane CENTRAL Register of Studies Online (CRSO), and The Virtual Health Library portal up to July 2020. We included papers that mentioned and/or described methods for screening or assessing research misconduct in health-related research. We categorized identified methods into the following four groups according to their scopes: overall concern, textual concern, image concern, and data concern.

RESULTS

We included 57 papers reporting on 27 methods: two on overall concern, four on textual concern, three on image concern, and 18 on data concern. Apart from the methods to locate textual plagiarism and image manipulation, all other methods, be it theoretical or empirical, are based on examples, are not standardized, and lack formal validation.

CONCLUSION

Existing methods cover a wide range of issues regarding research misconduct. Although measures to counteract textual plagiarism are well implemented, tools to investigate other forms of research misconduct are rudimentary and labour-intensive. To cope with the rising challenge of research misconduct, further development of automatic tools and routine validation of these methods is needed.

TRIAL REGISTRATION NUMBER

Center for Open Science (OSF) (https://osf.io/mq89w).

Benford's law and metabolomics: A tale of numbers and blood

The Newcomb-Benford law - also known as the "law of anomalous numbers" or, more commonly, Benford's law - predicts that the distribution of the first significant digit of random numbers obtained from mixed probability distributions follows a predictable pattern and reveals some universal behavior. Specifically, given a dataset of empirical measures, the likelihood of the first digit of any number being 1 is ∼30 %, ∼18 % for 2, 12.5 % for 3 and so on, with a decreasing probability all the way to number 9. If the digits were distributed uniformly, all the numbers 1 through 9 would have the same probability to appear as the first digit in any given empirical random measurement. However, this is not the case, as this law defies common sense and seems to apply seamlessly to large data. The use of omics technologies and, in particular, metabolomics has generated a wealth of big data in the field of transfusion medicine. In the present meta-analysis, we focused on previous big data from metabolomics studies of relevance to transfusion medicine: one on the quality of stored red blood cells, one on the phenotypes of transfusion recipients, i.e. trauma patients suffering from trauma and hemorrhage, and one of relevance to the 2020 SARS-COV-2 global pandemic. We show that metabolomics data follow a Benford's law distribution, an observation that could be relevant for future application of the "law of anomalous numbers" in the field of quality control processes in transfusion medicine.

Pinocchio testing in the forensic analysis of waiting lists: using public waiting list data from Finland and Spain for testing Newcomb-Benford's Law

OBJECTIVE

Newcomb-Benford's Law (NBL) proposes a regular distribution for first digits, second digits and digit combinations applicable to many different naturally occurring sources of data. Testing deviations from NBL is used in many datasets as a screening tool for identifying data trustworthiness problems. This study aims to compare public available waiting lists (WL) data from Finland and Spain for testing NBL as an instrument to flag up potential manipulation in WLs.

DESIGN

Analysis of the frequency of Finnish and Spanish WLs first digits to determine if their distribution is similar to the pattern documented by NBL. Deviations from the expected first digit frequency were analysed using Pearson's χ², mean absolute deviation and Kuiper tests.

SETTING/PARTICIPANTS

Publicly available WL data from Finland and Spain, two countries with universal health insurance and National Health Systems but characterised by different levels of transparency and good governance standards.

MAIN OUTCOME MEASURES

Adjustment of the observed distribution of the numbers reported in Finnish and Spanish WL data to the expected distribution according to NBL.

RESULTS

WL data reported by the Finnish health system fits first digit NBL according to all statistical tests used (p=0.6519 in χ² test). For Spanish data, this hypothesis was rejected in all tests (p<0.0001 in χ² test).

CONCLUSIONS

Testing deviations from NBL distribution can be a useful tool to identify problems with WL data trustworthiness and signalling the need for further testing.