Applying the Non-adoption, Abandonment, Scale-up, Spread and Sustainability (NASSS) framework to evaluate automated evidence synthesis in health behaviour change

Automated tools to speed up the process of evidence synthesis are increasingly apparent within health behaviour research. This brief review explores the potential of the Non-adoption, Abandonment, Scale-up, Spread and Sustainability framework for supporting automated evidence synthesis in health behaviour change by applying it to the ongoing Human Behaviour-Change Project, which aims to revolutionize evidence synthesis within behaviour change intervention research. To increase the relevance of NASSS for health behaviour change, we recommend i) terminology changes ('condition' to 'behaviour' and 'patient' to 'end user') and ii) that it is used prospectively address complexities iteratively. We draw conclusions about i) the need to specify the organizations that will use the technology, ii) identifying what to do if interdependencies fail and iii) even though we have focused on automated evidence synthesis, NASSS would arguably be beneficial for technology developments in health behaviour change more generally, particularly for invention development.

Using machine learning to extract information and predict outcomes from reports of randomised trials of smoking cessation interventions in the Human Behaviour-Change Project

Background  Using reports of randomised trials of smoking cessation interventions as a test case, this study aimed to develop and evaluate machine learning (ML) algorithms for extracting information from study reports and predicting outcomes as part of the Human Behaviour-Change Project. It is the first of two linked papers, with the second paper reporting on further development of a prediction system. Methods  Researchers manually annotated 70 items of information ('entities') in 512 reports of randomised trials of smoking cessation interventions covering intervention content and delivery, population, setting, outcome and study methodology using the Behaviour Change Intervention Ontology. These entities were used to train ML algorithms to extract the information automatically. The information extraction ML algorithm involved a named-entity recognition system using the 'FLAIR' framework. The manually annotated intervention, population, setting and study entities were used to develop a deep-learning algorithm using multiple layers of long-short-term-memory (LSTM) components to predict smoking cessation outcomes. Results  The F1 evaluation score, derived from the false positive and false negative rates (range 0-1), for the information extraction algorithm averaged 0.42 across different types of entity (SD=0.22, range 0.05-0.88) compared with an average human annotator's score of 0.75 (SD=0.15, range 0.38-1.00). The algorithm for assigning entities to study arms ( e.g., intervention or control) was not successful. This initial ML outcome prediction algorithm did not outperform prediction based just on the mean outcome value or a linear regression model. Conclusions  While some success was achieved in using ML to extract information from reports of randomised trials of smoking cessation interventions, we identified major challenges that could be addressed by greater standardisation in the way that studies are reported. Outcome prediction from smoking cessation studies may benefit from development of novel algorithms, e.g., using ontological information to inform ML (as reported in the linked paper 3).

Systematic reviews of convalescent plasma in COVID-19 continue to be poorly conducted and reported: a systematic review

OBJECTIVES

To suggest possible approaches to combatting the impact of the COVID-19 infodemic to prevent research waste in future health emergencies and in everyday research and practice.

STUDY DESIGN AND SETTING

Systematic review. The Epistemonikos database was searched in June 2021 for systematic reviews on the effectiveness of convalescent plasma for COVID-19. Two reviewers independently screened the retrieved references with disagreements resolved by discussion. Data extraction was completed by one reviewer with a proportion checked by a second. We used the Assessment of Multiple Systematic Reviews to assess the quality of conduct and reporting of included reviews.

RESULTS

Fifty one systematic reviews are included with 193 individual studies included within the systematic reviews. There was considerable duplication of effort; multiple reviews were conducted at the same time with inconsistencies in the evidence included. The reviews were of low methodological quality, poorly reported, and did not adhere to preferred reporting items for systematic reviews and meta-analysis guidance.

CONCLUSION

Researchers need to conduct, appraise, interpret, and disseminate systematic reviews better. All in the research community (researchers, peer-reviewers, journal editors, funders, decision makers, clinicians, journalists, and the public) need to work together to facilitate the conduct of robust systematic reviews that are published and communicated in a timely manner, reducing research duplication and waste, increasing transparency and accessibility of all systematic reviews.