Rapid review methods series: Guidance on the use of supportive software

This paper is part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group. Rapid reviews (RRs) use modified systematic review methods to accelerate the review process while maintaining systematic, transparent and reproducible methods. This paper guides how to use supportive software for RRs.We strongly encourage the use of supportive software throughout RR production. Specifically, we recommend (1) using collaborative online platforms that enable working in parallel, allow for real-time project management and centralise review details; (2) using automation software to support, but not entirely replace a human reviewer and human judgement and (3) being transparent in reporting the methodology and potential risk for bias due to the use of supportive software.

Machine learning reduced workload for the Cochrane COVID-19 Study Register: development and evaluation of the Cochrane COVID-19 Study Classifier

BACKGROUND

This study developed, calibrated and evaluated a machine learning (ML) classifier designed to reduce study identification workload in maintaining the Cochrane COVID-19 Study Register (CCSR), a continuously updated register of COVID-19 research studies.

METHODS

A ML classifier for retrieving COVID-19 research studies (the 'Cochrane COVID-19 Study Classifier') was developed using a data set of title-abstract records 'included' in, or 'excluded' from, the CCSR up to 18th October 2020, manually labelled by information and data curation specialists or the Cochrane Crowd. The classifier was then calibrated using a second data set of similar records 'included' in, or 'excluded' from, the CCSR between October 19 and December 2, 2020, aiming for 99% recall. Finally, the calibrated classifier was evaluated using a third data set of similar records 'included' in, or 'excluded' from, the CCSR between the 4th and 19th of January 2021.

RESULTS

The Cochrane COVID-19 Study Classifier was trained using 59,513 records (20,878 of which were 'included' in the CCSR). A classification threshold was set using 16,123 calibration records (6005 of which were 'included' in the CCSR) and the classifier had a precision of 0.52 in this data set at the target threshold recall >0.99. The final, calibrated COVID-19 classifier correctly retrieved 2285 (98.9%) of 2310 eligible records but missed 25 (1%), with a precision of 0.638 and a net screening workload reduction of 24.1% (1113 records correctly excluded).

CONCLUSIONS

The Cochrane COVID-19 Study Classifier reduces manual screening workload for identifying COVID-19 research studies, with a very low and acceptable risk of missing eligible studies. It is now deployed in the live study identification workflow for the Cochrane COVID-19 Study Register.

Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews

OBJECTIVES

This study developed, calibrated, and evaluated a machine learning classifier designed to reduce study identification workload in Cochrane for producing systematic reviews.

METHODS

A machine learning classifier for retrieving randomized controlled trials (RCTs) was developed (the "Cochrane RCT Classifier"), with the algorithm trained using a data set of title-abstract records from Embase, manually labeled by the Cochrane Crowd. The classifier was then calibrated using a further data set of similar records manually labeled by the Clinical Hedges team, aiming for 99% recall. Finally, the recall of the calibrated classifier was evaluated using records of RCTs included in Cochrane Reviews that had abstracts of sufficient length to allow machine classification.

RESULTS

The Cochrane RCT Classifier was trained using 280,620 records (20,454 of which reported RCTs). A classification threshold was set using 49,025 calibration records (1,587 of which reported RCTs), and our bootstrap validation found the classifier had recall of 0.99 (95% confidence interval 0.98-0.99) and precision of 0.08 (95% confidence interval 0.06-0.12) in this data set. The final, calibrated RCT classifier correctly retrieved 43,783 (99.5%) of 44,007 RCTs included in Cochrane Reviews but missed 224 (0.5%). Older records were more likely to be missed than those more recently published.

CONCLUSIONS

The Cochrane RCT Classifier can reduce manual study identification workload for Cochrane Reviews, with a very low and acceptable risk of missing eligible RCTs. This classifier now forms part of the Evidence Pipeline, an integrated workflow deployed within Cochrane to help improve the efficiency of the study identification processes that support systematic review production.

Cochrane's Linked Data Project: How it Can Advance our Understanding of Surrogate Endpoints

Cochrane has developed a linked data infrastructure to make the evidence and data from its rich repositories more discoverable to facilitate evidence-based health decision-making. These annotated resources can enhance the study and understanding of biomarkers and surrogate endpoints.

Facilitating Web-Based Collaboration in Evidence Synthesis (TaskExchange): Development and Analysis

BACKGROUND

The conduct and publication of scientific research are increasingly open and collaborative. There is growing interest in Web-based platforms that can effectively enable global, multidisciplinary scientific teams and foster networks of scientists in areas of shared research interest. Designed to facilitate Web-based collaboration in research evidence synthesis, TaskExchange highlights the potential of these kinds of platforms.

OBJECTIVE

This paper describes the development, growth, and future of TaskExchange, a Web-based platform facilitating collaboration in research evidence synthesis.

METHODS

The original purpose of TaskExchange was to create a platform that connected people who needed help with their Cochrane systematic reviews (rigorous syntheses of health research) with people who had the time and expertise to help. The scope of TaskExchange has now been expanded to include other evidence synthesis tasks, including guideline development. The development of TaskExchange was initially undertaken in 5 agile development phases with substantial user engagement. In each phase, software was iteratively deployed as it was developed and tested, enabling close cycles of development and refinement.

RESULTS

TaskExchange enables users to browse and search tasks and members by keyword or nested filters, post and respond to tasks, sign up to notification emails, and acknowledge the work of TaskExchange members. The pilot platform has been open access since August 2016, has over 2300 members, and has hosted more than 630 tasks, covering a wide range of research synthesis-related tasks. Response rates are consistently over 75%, and user feedback has been positive.

CONCLUSIONS

TaskExchange demonstrates the potential for new technologies to support Web-based collaboration in health research. Development of a relatively simple platform for peer-to-peer exchange has provided opportunities for systematic reviewers to get their reviews completed more quickly and provides an effective pathway for people to join the global health evidence community.

A taxonomy has been developed for outcomes in medical research to help improve knowledge discovery

Objectives

There is increasing recognition that insufficient attention has been paid to the choice of outcomes measured in clinical trials. The lack of a standardized outcome classification system results in inconsistencies due to ambiguity and variation in how outcomes are described across different studies. Being able to classify by outcome would increase efficiency in searching sources such as clinical trial registries, patient registries, the Cochrane Database of Systematic Reviews, and the Core Outcome Measures in Effectiveness Trials (COMET) database of core outcome sets (COS), thus aiding knowledge discovery.

Study Design and Setting

A literature review was carried out to determine existing outcome classification systems, none of which were sufficiently comprehensive or granular for classification of all potential outcomes from clinical trials. A new taxonomy for outcome classification was developed, and as proof of principle, outcomes extracted from all published COS in the COMET database, selected Cochrane reviews, and clinical trial registry entries were classified using this new system.

Results

Application of this new taxonomy to COS in the COMET database revealed that 274/299 (92%) COS include at least one physiological outcome, whereas only 177 (59%) include at least one measure of impact (global quality of life or some measure of functioning) and only 105 (35%) made reference to adverse events.

Conclusions

This outcome taxonomy will be used to annotate outcomes included in COS within the COMET database and is currently being piloted for use in Cochrane Reviews within the Cochrane Linked Data Project. Wider implementation of this standard taxonomy in trial and systematic review databases and registries will further promote efficient searching, reporting, and classification of trial outcomes.

Living systematic reviews: 2. Combining human and machine effort

New approaches to evidence synthesis, which use human effort and machine automation in mutually reinforcing ways, can enhance the feasibility and sustainability of living systematic reviews. Human effort is a scarce and valuable resource, required when automation is impossible or undesirable, and includes contributions from online communities ("crowds") as well as more conventional contributions from review authors and information specialists. Automation can assist with some systematic review tasks, including searching, eligibility assessment, identification and retrieval of full-text reports, extraction of data, and risk of bias assessment. Workflows can be developed in which human effort and machine automation can each enable the other to operate in more effective and efficient ways, offering substantial enhancement to the productivity of systematic reviews. This paper describes and discusses the potential-and limitations-of new ways of undertaking specific tasks in living systematic reviews, identifying areas where these human/machine "technologies" are already in use, and where further research and development is needed. While the context is living systematic reviews, many of these enabling technologies apply equally to standard approaches to systematic reviewing.

Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap

The current difficulties in keeping systematic reviews up to date leads to considerable inaccuracy, hampering the translation of knowledge into action. Incremental advances in conventional review updating are unlikely to lead to substantial improvements in review currency. A new approach is needed. We propose living systematic review as a contribution to evidence synthesis that combines currency with rigour to enhance the accuracy and utility of health evidence. Living systematic reviews are high quality, up-to-date online summaries of health research, updated as new research becomes available, and enabled by improved production efficiency and adherence to the norms of scholarly communication. Together with innovations in primary research reporting and the creation and use of evidence in health systems, living systematic review contributes to an emerging evidence ecosystem.

Long-acting beta2-agonists for chronic obstructive pulmonary disease

BACKGROUND

Chronic obstructive pulmonary disease (COPD) is a respiratory disease that causes progressive symptoms of breathlessness, cough and mucus build-up. It is the fourth or fifth most common cause of death worldwide and is associated with significant healthcare costs.Inhaled long-acting beta2-agonists (LABAs) are widely prescribed to manage the symptoms of COPD when short-acting agents alone are no longer sufficient. Twice-daily treatment with an inhaled LABA is aimed at relieving symptoms, improving exercise tolerance and quality of life, slowing decline and even improving lung function and preventing and treating exacerbations.

OBJECTIVES

To assess the effects of twice-daily long-acting beta2-agonists compared with placebo for patients with COPD on the basis of clinically important endpoints, primarily quality of life and COPD exacerbations.

SEARCH METHODS

We searched the Cochrane Airways Group trials register, ClinicalTrials.gov and manufacturers' websites in June 2013.

SELECTION CRITERIA

Parallel, randomised controlled trials (RCTs) recruiting populations of patients with chronic obstructive pulmonary disease. Studies were required to be at least 12 weeks in duration and designed to assess the safety and efficacy of a long-acting beta2-agonist against placebo.

DATA COLLECTION AND ANALYSIS

Data and characteristics were extracted independently by two review authors, and each study was assessed for potential sources of bias. Data for all outcomes were pooled and subgrouped by LABA agent (formoterol 12 μg, formoterol 24 μg and salmeterol 50 μg) and then were separately analysed by LABA agent and subgrouped by trial duration. Sensitivity analyses were conducted for the proportion of participants taking inhaled corticosteroids and for studies with high or uneven rates of attrition.

MAIN RESULTS

Twenty-six RCTs met the inclusion criteria, randomly assigning 14,939 people with COPD to receive twice-daily LABA or placebo. Study duration ranged from three months to three years; the median duration was six months. Participants were more often male with moderate to severe symptoms at randomisation; mean forced expiratory volume in 1 second (FEV1) was between 33% and 55% predicted normal in the studies, and mean St George's Respiratory Questionnaire score (SGRQ) ranged from 44 to 55 when reported.Moderate-quality evidence showed that LABA treatment improved quality of life on the SGRQ (mean difference (MD) -2.32, 95% confidence interval (CI) -3.09 to -1.54; I(2) = 50%; 17 trials including 11,397 people) and reduced the number of exacerbations requiring hospitalisation (odds ratio (OR) 0.73, 95% CI 0.56 to 0.95; I(2) = 10%; seven trials including 3804 people). In absolute terms, 18 fewer people per 1000 were hospitalised as the result of an exacerbation while receiving LABA therapy over a weighted mean of 7 months (95% CI 3 to 31 fewer). Scores were also improved on the Chronic Respiratory Disease Questionnaire (CRQ), and more people receiving LABA treatment showed clinically important improvement of at least four points on the SGRQ.The number of people who had exacerbations requiring a course of oral steroids or antibiotics was also lower among those taking LABA (52 fewer per 1000 treated over 8 months; 95% CI 24 to 78 fewer, moderate quality evidence).Mortality was low, and combined findings of all studies showed that LABA therapy did not significantly affect mortality (OR 0.90, 95% CI 0.75 to 1.08; I(2) = 21%; 23 trials including 14,079 people, moderate quality evidence). LABA therapy did not affect the rate of serious adverse events (OR 0.97, 95% CI 0.83 to 1.14; I(2) = 34%, moderate quality evidence), although there was significant unexplained heterogeneity, especially between the two formoterol doses.LABA therapy improved predose FEV1 by 73 mL more than placebo (95% CI 48 to 98; I(2) = 71%, low quality evidence), and people were more likely to withdraw from placebo than from LABA therapy (OR 0.74, 95% CI 0.69 to 0.80; I(2) = 0%). Higher rates of withdrawal in the placebo arm may reduce our confidence in some results, but the disparity is more likely to reduce the magnitude of difference between LABA and placebo than inflate the true effect; removing studies at highest risk of bias on the basis of high and unbalanced attrition did not change conclusions for the primary outcomes.

AUTHORS' CONCLUSIONS

Moderate-quality evidence from 26 studies showed that inhaled long-acting beta2-agonists are effective over the medium and long term for patients with moderate to severe COPD. Their use is associated with improved quality of life and reduced exacerbations, including those requiring hospitalisation. Overall, findings showed that inhaled LABAs did not significantly reduce mortality or serious adverse events.