Replicability, Reproducibility, and Fragility of Research Findings—Ultimately, Caveat Emptor

Refining the CRiSPHe (checklist for reporting research using a simulated patient methodology in Health): a Delphi study

OBJECTIVES

A recent applicability study highlighted the need for the existing checklist for reporting research using a simulated patient methodology (CRiSP) to be clearer and user-friendly. The aim of this study was to update the checklist to address these concerns.

METHODS

A fourth round of the Delphi consensus study, used in the original checklist development work, was conducted. Previous participants, who had expertise in SP methodology, were invited to complete a questionnaire including a list of 13 checklist items developed in the previous study and revised following applicability testing. Closed questions were analysed for frequency. Consensus was predefined as >80% agreement. All items were discussed in a roundtable meeting and further modified as necessary. Responses to open questions were content analysed.

KEY FINDINGS

Twenty-one authors participated. There was a statistical consensus in 12 out of 13 modified checklist items.

CONCLUSIONS

A final reporting checklist for studies in health research using SP methodology has been developed using a consensus approach. Further refinements may be needed to increase the generalizability of the checklist in different contexts.

Narrative Review of Statistical Reporting Checklists, Mandatory Statistical Editing, and Rectifying Common Problems in the Reporting of Scientific Articles

Considerable attention has been drawn to poor reproducibility in the biomedical literature. One explanation is inadequate reporting of statistical methods by authors and inadequate assessment of statistical reporting and methods during peer review. In this narrative review, we examine scientific studies of several well-publicized efforts to improve statistical reporting. We also review several retrospective assessments of the impact of these efforts. These studies show that instructions to authors and statistical checklists are not sufficient; no findings suggested that either improves the quality of statistical methods and reporting. Second, even basic statistics, such as power analyses, are frequently missing or incorrectly performed. Third, statistical review is needed for all papers that involve data analysis. A consistent finding in the studies was that nonstatistical reviewers (eg, "scientific reviewers") and journal editors generally poorly assess statistical quality. We finish by discussing our experience with statistical review at Anesthesia & Analgesia from 2006 to 2016.