Evidence appraisal: a scoping review, conceptual framework, and research agenda

PubMed captures more fine-grained bibliographic data on scientific commentary than Web of Science: a comparative analysis

BACKGROUND

Research commentaries have the potential for evidence appraisal in emphasising, correcting, shaping and disseminating scientific knowledge.

OBJECTIVES

To identify the appropriate bibliographic source for capturing commentary information, this study compares comment data in PubMed and Web of Science (WoS) to assess their applicability in evidence appraisal.

METHODS

Using COVID-19 as a case study, with over 27 k COVID-19 papers in PubMed as a baseline, we designed a comparative analysis for commented-commenting relations in two databases from the same dataset pool, making a fair and reliable comparison. We constructed comment networks for each database for network structural analysis and compared the characteristics of commentary materials and commented papers from various facets.

RESULTS

For network comparison, PubMed surpasses WoS with more closed feedback loops, reaching a deeper six-level network compared with WoS' four levels, making PubMed well-suited for evidence appraisal through argument mining. PubMed excels in identifying specialised comments, displaying significantly lower author count (mean, 3.59) and page count (mean, 1.86) than WoS (authors, 4.31, 95% CI of difference of two means = [0.66, 0.79], p<0.001; pages, 2.80, 95% CI of difference of two means = [0.87, 1.01], p<0.001), attributed to PubMed's CICO comment identification algorithm. Commented papers in PubMed also demonstrate higher citations and stronger sentiments, especially significantly elevated disputed rates (PubMed, 24.54%; WoS, 18.8%; baseline, 8.3%; all p<0.0001). Additionally, commented papers in both sources exhibit superior network centrality metrics compared with WoS-only counterparts.

CONCLUSION

Considering the impact and controversy of commented works, the accuracy of comments and the depth of network interactions, PubMed potentially serves as a valuable resource in evidence appraisal and detection of controversial issues compared with WoS.

Retrieval augmented scientific claim verification

Objective

To automate scientific claim verification using PubMed abstracts.

Materials and Methods

We developed CliVER, an end-to-end scientific Claim VERification system that leverages retrieval-augmented techniques to automatically retrieve relevant clinical trial abstracts, extract pertinent sentences, and use the PICO framework to support or refute a scientific claim. We also created an ensemble of three state-of-the-art deep learning models to classify rationale of support, refute, and neutral. We then constructed CoVERt, a new COVID VERification dataset comprising 15 PICO-encoded drug claims accompanied by 96 manually selected and labeled clinical trial abstracts that either support or refute each claim. We used CoVERt and SciFact (a public scientific claim verification dataset) to assess CliVER's performance in predicting labels. Finally, we compared CliVER to clinicians in the verification of 19 claims from 6 disease domains, using 189 648 PubMed abstracts extracted from January 2010 to October 2021.

Results

In the evaluation of label prediction accuracy on CoVERt, CliVER achieved a notable F1 score of 0.92, highlighting the efficacy of the retrieval-augmented models. The ensemble model outperforms each individual state-of-the-art model by an absolute increase from 3% to 11% in the F1 score. Moreover, when compared with four clinicians, CliVER achieved a precision of 79.0% for abstract retrieval, 67.4% for sentence selection, and 63.2% for label prediction, respectively.

Conclusion

CliVER demonstrates its early potential to automate scientific claim verification using retrieval-augmented strategies to harness the wealth of clinical trial abstracts in PubMed. Future studies are warranted to further test its clinical utility.

AI-generated text may have a role in evidence-based medicine

E vidence-based medicine (EBM) requires the retrieval and ranking of relevant evidence by epistemological strength, to identify the most appropriate evidence to inform guidelines and policies, with a preference for robust evidence from randomized clinical trials (RCTs), systematic reviews, and meta-analyses. The explosive growth of the scientific literature and the emergence of new sources of evidence, including social media, case reports, and large-scale observational studies, as well as the free-text nature of this large body of evidence, collectively make it difficult to appraise and select the best available evidence.

A comment-driven evidence appraisal approach to promoting research findings into practice when only uncertain evidence is available

BACKGROUND

Comments in PubMed are usually short papers for supporting or refuting claims, or discussing methods and findings in original articles. This study aims to explore whether they can be used as a quick and reliable evidence appraisal instrument for promoting research findings into practice, especially in emergency situations such as COVID-19 in which only missing, incomplete or uncertain evidence is available.

METHODS

Evidence-comment networks (ECNs) were constructed by linking COVID-19-related articles to the commentaries (letters, editorials or brief correspondence) they received. PubTator Central was used to extract entities with a high volume of comments from the titles and abstracts of the articles. Among them, six drugs were selected, and their evidence assertions were analysed by exploring the structural information in the ECNs as well as the sentiment of the comments (positive, negative, neutral). Recommendations in WHO guidelines were used as the gold standard control to validate the consistency, coverage and efficiency of comments in reshaping clinical knowledge claims.

RESULTS

The overall positive/negative sentiments of comments were aligned with recommendations for/against the corresponding treatments in the WHO guidelines. Comment topics covered all significant points of evidence appraisal and beyond. Furthermore, comments may indicate the uncertainty regarding drug use for clinical practice. Half of the critical comments emerged 4.25 months earlier on average than the guideline release.

CONCLUSIONS

Comments have the potential as a support tool for rapid evidence appraisal as they have a selection effect by appraising the benefits, limitations and other clinical practice issues of concern in existing evidence. We suggest as a future direction an appraisal framework based on the comment topics and sentiment orientations to leverage the potential of scientific commentaries supporting evidence appraisal and decision-making.

The voices of children and young people during COVID-19: A critical review of methods

AIM

Critically review research methods used to elicit children and young people's views and experiences in the first year of COVID-19, using an ethical and child rights lens.

METHODS

A systematic search of peer-reviewed literature on children and young people's perspectives and experiences of COVID-19. LEGEND (Let Evidence Guide Every New Decision) tools were applied to assess the quality of included studies. The critical review methodology addressed four ethical parameters: (1) Duty of care; (2) Children and young people's consent; (3) Communication of findings; and (4) Reflexivity.

RESULTS

Two phases of searches identified 8131 studies; 27 studies were included for final analysis, representing 43,877 children and young people's views. Most studies were from high-income countries. Three major themes emerged: (a) Whose voices are heard; (b) How are children and young people heard; and (c) How do researchers engage in reflexivity and ethical practice? Online surveys of children and young people from middle-class backgrounds dominated the research during COVID-19. Three studies actively involved children and young people in the research process; two documented a rights-based framework. There was limited attention paid to some ethical issues, particularly the lack of inclusion of children and young people in research processes.

CONCLUSION

There are equity gaps in accessing the experiences of children and young people from disadvantaged settings. Most children and young people were not involved in shaping research methods by soliciting their voices.

A neuro-symbolic method for understanding free-text medical evidence

OBJECTIVE

We introduce Medical evidence Dependency (MD)-informed attention, a novel neuro-symbolic model for understanding free-text clinical trial publications with generalizability and interpretability.

MATERIALS AND METHODS

We trained one head in the multi-head self-attention model to attend to the Medical evidence Ddependency (MD) and to pass linguistic and domain knowledge on to later layers (MD informed). This MD-informed attention model was integrated into BioBERT and tested on 2 public machine reading comprehension benchmarks for clinical trial publications: Evidence Inference 2.0 and PubMedQA. We also curated a small set of recently published articles reporting randomized controlled trials on COVID-19 (coronavirus disease 2019) following the Evidence Inference 2.0 guidelines to evaluate the model's robustness to unseen data.

RESULTS

The integration of MD-informed attention head improves BioBERT substantially in both benchmark tasks-as large as an increase of +30% in the F1 score-and achieves the new state-of-the-art performance on the Evidence Inference 2.0. It achieves 84% and 82% in overall accuracy and F1 score, respectively, on the unseen COVID-19 data.

CONCLUSIONS

MD-informed attention empowers neural reading comprehension models with interpretability and generalizability via reusable domain knowledge. Its compositionality can benefit any transformer-based architecture for machine reading comprehension of free-text medical evidence.

Learning to Critically Appraise Rheumatic Disease Literature: Educational Opportunities During Training and into Practice

To provide optimal patient care, rheumatologists must be equipped and motivated to critically appraise the literature. The conceptual frameworks Retrieval Enhanced Learning, Self-Determination Theory, and Communities of Practice can inform the design of educational approaches to promote critical appraisal in practice. HSS CLASS-Rheum® is a learning tool that can be used to help rheumatologists learn skills for critical appraisal through retrieval practice. Combining retrieval practice with opportunities for connection through Peer Instruction, journal clubs, and other forums can help support engagement and internalization of motivation, promoting persistence with critical appraisal in practice.

Preparing and Presenting Journal Club Content: An Essential Component of Homeopathic Learning

Homeopathic education has passed through significant incremental changes in the past few years, where especially postgraduate education has become increasingly slanted toward advanced knowledge of clinical work and research methods. Among many educational activities, a great source of learning is from presenting at or attending a journal club meeting, which is a gathering of people to learn and to critically appraise a journal article or other study material. There has been little previous guidance in homeopathy regarding how to prepare and present journal club content. Selection of a suitable topic is one of the critical prerequisites. Each and every step, from preparation to presentation, needs to be carefully planned and considered. For the meeting to be successful, the final discussion phase requires the active participation and critical insight of all those attending.

Understanding the nature and scope of clinical research commentaries in PubMed

Scientific commentaries are expected to play an important role in evidence appraisal, but it is unknown whether this expectation has been fulfilled. This study aims to better understand the role of scientific commentary in evidence appraisal. We queried PubMed for all clinical research articles with accompanying comments and extracted corresponding metadata. Five percent of clinical research studies (N = 130 629) received postpublication comments (N = 171 556), resulting in 178 882 comment-article pairings, with 90% published in the same journal. We obtained 5197 full-text comments for topic modeling and exploratory sentiment analysis. Topics were generally disease specific with only a few topics relevant to the appraisal of studies, which were highly prevalent in letters. Of a random sample of 518 full-text comments, 67% had a supportive tone. Based on our results, published commentary, with the exception of letters, most often highlight or endorse previous publications rather than serve as a prominent mechanism for critical appraisal.

Analysis of evidence appraisals for interventional studies in family medicine using an informatics approach

This study reports the first assessment of published comments in the family medicine literature using structured codes, which produced commentary annotations that will be the foundation of a knowledge base of appraisals of family medicine trials. Evidence appraisal occurs in a variety of formats and serves to shed light on the quality of research. However, scientific discourse generally and evidence appraisal in particular has not itself been analyzed for insights. A search strategy was devised to identify all journal comments indexed in PubMed linked to controlled intervention studies published in a recent 15-year period in major family medicine journals. A previously developed structured representation in the form of a list of appraisal concepts was used to formally annotate and categorize the journal comments through an iterative process. Trends in family medicine evidence appraisal were then analyzed. A total of 93 comments on studies from five journals over 15 years were included in the analysis. Two thirds of extracted appraisals were negative criticisms. All appraisals of measurement instruments were negative (100%). The participants baseline characteristics, the author discussions, and the design of the interventions were also criticized (respectively 91.7%, 84.6% and 83.3% negative). In contrast, appraisals of the scientific basis of the studies were positive (81.8%). The categories with the most appraisals were, most generally, those focused on the study design, and most specifically, those focused on the scientific basis. This study provides a new data-driven approach to review scientific discourse regarding the strengths and limitations of research within academic family medicine. This methodology can potentially generalize to other medical domains. Structured appraisal data generated here will enable future clinical, scientific, and policy decision-making and broader meta-research in family medicine.

Construction of a Different Polymer Chain Structure to Study π-π Interaction between Polymer and Reduced Graphene Oxide

In this work, a different polymer chain structure was synthesized to study π-π interactions between polymer and reduced graphene oxide (RGO). Polymers with different chain structures were obtained from free radical copolymerization of styrene with 4-cyanostyrene (containing substituted phenyl rings) and 2-vinylnaphthalene (containing naphthalene rings). In this work, the polystyrene, poly(styrene-co-4-cyanostyrene) and poly(styrene-co-2-vinylnaphthalene) were named as PS, PSCN and PSNP, respectively. RGO was prepared through modified Hummers' method and further thermal reduction, and nanocomposites were prepared by solution blending. Thus, different π-π interactions were formed between polymers and RGO. Raman and thermal gravimetric analysis (TGA) were used to characterize the interfacial interaction, showing that the trend of the interfacial interaction should be in the order of RGO/PSCN, RGO/PS, and RGO/PSNP. The differential scanning calorimetry (DSC) measurement showed that, compared with polymer matrix, the glass transition temperature (Tg) of RGO/PS, RGO/PSCN and RGO/PSNP nanocomposites with the addition of 4.0 wt% RGO are increased by 14.3 °C, 25.2 °C and 4.4 °C, respectively. Compared with π-π interaction only formed through aromatic rings, substituent groups changed the densities of electron clouds on the phenyl rings. This change resulted in the formation of donor-acceptor interaction and reinforcement of the π-π interaction at the interface, which leads to increased value of Tg. This comparative study can be useful for selecting appropriate interaction groups, as well as suitable monomers, to prepare high performance nanocomposites.