Systematic review finds "spin" practices and poor reporting standards in studies on machine learning-based prediction models

OBJECTIVES

We evaluated the presence and frequency of spin practices and poor reporting standards in studies that developed and/or validated clinical prediction models using supervised machine learning techniques.

STUDY DESIGN AND SETTING

We systematically searched PubMed from 01/2018 to 12/2019 to identify diagnostic and prognostic prediction model studies using supervised machine learning. No restrictions were placed on data source, outcome, or clinical specialty.

RESULTS

We included 152 studies: 38% reported diagnostic models and 62% prognostic models. When reported, discrimination was described without precision estimates in 53/71 abstracts (74.6% [95% CI 63.4-83.3]) and 53/81 main texts (65.4% [95% CI 54.6-74.9]). Of the 21 abstracts that recommended the model to be used in daily practice, 20 (95.2% [95% CI 77.3-99.8]) lacked any external validation of the developed models. Likewise, 74/133 (55.6% [95% CI 47.2-63.8]) studies made recommendations for clinical use in their main text without any external validation. Reporting guidelines were cited in 13/152 (8.6% [95% CI 5.1-14.1]) studies.

CONCLUSION

Spin practices and poor reporting standards are also present in studies on prediction models using machine learning techniques. A tailored framework for the identification of spin will enhance the sound reporting of prediction model studies.

A randomized trial of an editorial intervention to reduce spin in the abstract's conclusion of manuscripts showed no significant effect

OBJECTIVE

To estimate the effect of an intervention compared to the usual peer-review process on reducing spin in the abstract's conclusion of biomedical study reports.

STUDY DESIGN AND SETTING

We conducted a two-arm, parallel-group RCT in a sample of primary research manuscripts submitted to BMJ Open. The authors received short instructions alongside the peer reviewers' comments in the intervention group. We assessed the presence of spin (primary outcome), types of spin, and wording change in the revised abstract's conclusion. Outcome assessors were blinded to the intervention assignment.

RESULTS

Of the 184 manuscripts randomized, 108 (54 intervention, 54 control) were selected for revision and could be evaluated for the presence of spin. The proportion of manuscripts with spin was 6% lower (95% CI: 24% lower to 13% higher) in the intervention group (57%, 31/54) than in the control group (63%, 34/54). The wording of the revised abstract's conclusion was changed in 34/54 (63%) manuscripts in the intervention group and 26/54 (48%) in the control group. The four prespecified types of spin involved (i) selective reporting (12 in the intervention group vs. 8 in the control group), (ii) including information not supported by evidence (9 vs. 9), and (iii) interpretation not consistent with the study results (14 vs. 18), and (iv) unjustified recommendations for practice (5 vs. 11).

CONCLUSION

These short instructions to authors did not have a statistically significant effect on reducing spin in revised abstract conclusions, and based on the confidence interval, the existence of a large effect can be excluded. Other interventions to reduce spin in reports of original research should be evaluated.

STUDY REGISTRATION

osf.io/xnuyt.