Development of a checklist for people communicating evidence-based information about the effects of healthcare interventions: a mixed methods study

OBJECTIVES

To make informed decisions about healthcare, patients and the public, health professionals and policymakers need information about the effects of interventions. People need information that is based on the best available evidence; that is presented in a complete and unbiased way; and that is relevant, trustworthy and easy to use and to understand. The aim of this paper is to provide guidance and a checklist to those producing and communicating evidence-based information about the effects of interventions intended to inform decisions about healthcare.

DESIGN

To inform the development of this checklist, we identified research relevant to communicating evidence-based information about the effects of interventions. We used an iterative, informal consensus process to synthesise our recommendations. We began by discussing and agreeing on some initial recommendations, based on our own experience and research over the past 20-30 years. Subsequent revisions were informed by the literature we examined and feedback. We also compared our recommendations to those made by others. We sought structured feedback from people with relevant expertise, including people who prepare and use information about the effects of interventions for the public, health professionals or policymakers.

RESULTS

We produced a checklist with 10 recommendations. Three recommendations focus on making it easy to quickly determine the relevance of the information and find the key messages. Five recommendations are about helping the reader understand the size of effects and how sure we are about those estimates. Two recommendations are about helping the reader put information about intervention effects in context and understand if and why the information is trustworthy.

CONCLUSIONS

These 10 recommendations summarise lessons we have learnt developing and evaluating ways of helping people to make well-informed decisions by making research evidence more understandable and useful for them. We welcome feedback for how to improve our advice.

Establishing a taxonomy of potential hazards associated with communicating medical science in the age of disinformation

OBJECTIVES

Disinformation on medical matters has become an increasing public health concern. Public engagement by scientists, clinicians and patient advocates can contribute towards public understanding of medicine. However, depth of feeling on many issues (notably vaccination and cancer) can lead to adverse reactions for those communicating medical science, including vexatious interactions and targeted campaigns. Our objective in this work is to establish a taxonomy of common negative experiences encountered by those communicating medical science, and suggest guidelines so that they may be circumvented.

DESIGN

We establish a taxonomy of the common negative experiences reported by those communicating medical science, informed by surveying medical science communicators with public platforms.

PARTICIPANTS

142 prominent medical science communicators (defined as having >1000 Twitter followers and experience communicating medical science on social and traditional media platforms) were invited to take part in a survey, with 101 responses.

RESULTS

101 responses were analysed. Most participants experienced abusive behaviour (91.9%), including persistent harassment (69.3%) and physical violence and intimidation (5.9%). A substantial number (38.6%) received vexatious complaints to their employers, professional bodies or legal intimidation. The majority (62.4%) reported negative mental health sequelae due to public outreach, including depression, anxiety and stress. A significant minority (19.8%) were obligated to seek police advice or legal counsel due to actions associated with their outreach work. While the majority targeted with vexatious complaints felt supported by their employer/professional body, 32.4% reported neutral, poor or non-existent support.

CONCLUSIONS

Those engaging in public outreach of medical science are vulnerable to negative repercussions, and we suggest guidelines for professional bodies and organisations to remedy some of these impacts on front-line members.

An Acknowledgement to the HCA Healthcare Journal of Medicine's Initial Cohort of Reviewers, Authors, Editors and Advisory Board

Description As the HCA Healthcare Journal of Medicine reaches the half way point of our first year in publication, we want to thank those who have provided invaluable support for the journal since the first submissions arrived in November of last year. The successful launch of our journal would not have been possible without the assistance of our reviewers, authors and board members.

Effect of an editorial intervention to improve the completeness of reporting of randomised trials: a randomised controlled trial

OBJECTIVE

To evaluate the impact of an editorial intervention to improve completeness of reporting of reports of randomised trials.

DESIGN

Randomised controlled trial (RCT).

SETTING

BMJ Open's quality improvement programme.

PARTICIPANTS

24 manuscripts describing RCTs.

INTERVENTIONS

We used an R Shiny application to randomise manuscripts (1:1 allocation ratio, blocks of 4) to the intervention (n=12) or control (n=12) group. The intervention was performed by a researcher with expertise in the content of the Consolidated Standards of Reporting Trials (CONSORT) and consisted of an evaluation of completeness of reporting of eight core CONSORT items using the submitted checklist to locate information, and the production of a report containing specific requests for authors based on the reporting issues found, provided alongside the peer review reports. The control group underwent the usual peer review.

OUTCOMES

The primary outcome is the number of adequately reported items (0-8 scale) in the revised manuscript after the first round of peer review. The main analysis was intention-to-treat (n=24), and we imputed the scores of lost to follow-up manuscripts (rejected after peer review and not resubmitted). The secondary outcome is the proportion of manuscripts where each item was adequately reported. Two blinded reviewers assessed the outcomes independently and in duplicate and solved disagreements by consensus. We also recorded the amount of time to perform the intervention.

RESULTS

Manuscripts in the intervention group (mean: 7.01; SD: 1.47) were more completely reported than those in the control group (mean: 5.68; SD: 1.43) (mean difference 1.43, 95% CI 0.31 to 2.58). We observed the main differences in items 6a (outcomes), 9 (allocation concealment mechanism), 11a (blinding) and 17a (outcomes and estimation). The mean time to perform the intervention was 87 (SD 42) min.

CONCLUSIONS

We demonstrated the benefit of involving a reporting guideline expert in the editorial process. Improving the completeness of RCTs is essential to enhance their usability.

TRIAL REGISTRATION NUMBER

NCT03751878.

Defining predatory journals and responding to the threat they pose: a modified Delphi consensus process

OBJECTIVE

To conduct a Delphi survey informing a consensus definition of predatory journals and publishers.

DESIGN

This is a modified three-round Delphi survey delivered online for the first two rounds and in-person for the third round. Questions encompassed three themes: (1) predatory journal definition; (2) educational outreach and policy initiatives on predatory publishing; and (3) developing technological solutions to stop submissions to predatory journals and other low-quality journals.

PARTICIPANTS

Through snowball and purposive sampling of targeted experts, we identified 45 noted experts in predatory journals and journalology. The international group included funders, academics and representatives of academic institutions, librarians and information scientists, policy makers, journal editors, publishers, researchers involved in studying predatory journals and legitimate journals, and patient partners. In addition, 198 authors of articles discussing predatory journals were invited to participate in round 1.

RESULTS

A total of 115 individuals (107 in round 1 and 45 in rounds 2 and 3) completed the survey on predatory journals and publishers. We reached consensus on 18 items out of a total of 33 to be included in a consensus definition of predatory journals and publishers. We came to consensus on educational outreach and policy initiatives on which to focus, including the development of a single checklist to detect predatory journals and publishers, and public funding to support research in this general area. We identified technological solutions to address the problem: a 'one-stop-shop' website to consolidate information on the topic and a 'predatory journal research observatory' to identify ongoing research and analysis about predatory journals/publishers.

CONCLUSIONS

In bringing together an international group of diverse stakeholders, we were able to use a modified Delphi process to inform the development of a definition of predatory journals and publishers. This definition will help institutions, funders and other stakeholders generate practical guidance on avoiding predatory journals and publishers.

Make quotations great again: a proposal to reduce false-knowledge

The last decades saw remarkable change in the way healthcare professionals generate and consume medical knowledge. Information management technologies have evolved considerably, yet medical publications continue to use a referencing system that has changed very little since the turn of the 20th century. Research suggests that up to one in five referenced claims quotes the original text inaccurately. Many authors, perhaps inadvertently, contribute to this process by citing non-primary data and amplifying the errors of their predecessors. Erroneous claims are propagated, accumulate into false belief systems and generate inaccurate knowledge. Updating the referencing system to provide additional information to support each referenced claim (eg, the location of the referenced statement in the original text and the nature of that text) could, perhaps, address this cycle of inaccuracy. We believe such changes in the referencing system would prompt authors to rigorously verify referenced claims and provide readers with context to inform a critical evaluation of the text. We detail our proposal for changes in the notations used for referencing, as well as in the information provided within reference lists. We also discuss some barriers and solutions to the adoption of our proposal.

Associations Between Characteristics of Web-Based Diabetes News and Readers' Sentiments: Observational Study in the Netherlands

Background

Although experts agree that Web-based health information often contains exaggeration and misrepresentation of science, it is not yet known how this information affects the readers’ sentiments.

Objective

This study aimed to investigate whether specific aspects of Web-based diabetes research news are associated with positive or negative sentiments in readers.

Methods

A retrospective observational study of the comments on diabetes research news posted on Facebook pages was conducted as a function of the innovations’ developmental phase, the intended treatment effect, and the use of strong language to intensify the news messages (superlatives). Data for the investigation were drawn from the diabetes research news posted between January 2014 and January 2018 on the two largest Dutch Facebook pages on diabetes and the corresponding reader comments. By manually coding these Facebook user comments, three binary outcome variables were created, reflecting the presence of a positive sentiment, the presence of a negative sentiment, and the presence of a statement expressing hopefulness.

Results

Facebook users made a total of 3710 comments on 173 diabetes research news posts that were eligible for further analysis. Facebook user comments on posts about diabetes prevention (odds ratio [OR] 0.55, 95% CI 0.37-0.84), improved blood glucose regulation (OR 0.68, 95% CI 0.56-0.84), and symptom relief (OR 0.31, 95% CI 0.21-0.44) were associated with less positive sentiments as compared with potential diabetes cures. Furthermore, comments on innovations supported by preclinical evidence in animals were associated with more positive sentiments (OR 1.46, 95% CI 1.07-1.99) and statements expressing hope (OR 1.47, 95% CI 1.01-2.14), when compared with innovations that have evidence from large human trials. This study found no evidence for the associations between language intensification of the news posts and the readers’ sentiments.

Conclusions

Our finding that the attitudes toward diabetes research news on Facebook are most positive when clinical efficacy is not (or not yet) proven in large patient trials suggests that news authors and editors, as well as medical professionals, must exercise caution when acting as a conduit for diabetes research news.

Canadian French and English newspapers' portrayals of physicians' role and medical assistance in dying (MAiD) from 1972 to 2016: a qualitative textual analysis

OBJECTIVE

To examine how Canadian newspapers portrayed physicians' role and medical assistance in dying (MAiD).

DESIGN

Qualitative textual analysis.

SETTING

Online and print articles from Canadian French and English newspapers.

PARTICIPANTS

813 newspaper articles published from 1972 to 2016.

RESULTS

Key Canadian events defined five eras. From 1972 to 1990, newspapers portrayed physician's MAiD role as a social issue by reporting supportive public opinion polls and revealing it was already occurring in secret. From 1991 to 1995, newspapers discussed legal aspects of physicians' MAiD role including Rodriguez' Supreme Court of Canada appeal and Federal government Bills. From 1996 to 2004, journalists discussed professional aspects of physicians' MAiD role and the growing split between palliative care and physicians who supported MAiD. They also reported on court cases against Canadian physicians, Dr Kevorkian and suffering patients who could not receive MAiD. From 2005 to 2013, newspapers described political aspects including the tabling of MAiD legislation to change physicians' role. Lastly, from 2014 to 2016, newspapers again portrayed legal aspects of physicians' role as the Supreme Court of Canada was anticipated to legalise MAiD and the Québec government passed its own legislation. Remarkably, newspapers kept attention to MAiD over 44 years before it became legal. Articles generally reflected Canadians' acceptance of MAiD and physicians were typically portrayed as opposing it, but not all did.

CONCLUSIONS

Newspaper portrayals of physicians' MAiD role discussed public opinion, politicians' activities and professional and legal aspects. Portrayals followed the issue-attention cycle through three of five stages: 1) preproblem, 2) alarmed discovery and euphoric enthusiasm and 3) realising the cost of significant progress.

Effect of different financial competing interest statements on readers' perceptions of clinical educational articles: a randomised controlled trial

OBJECTIVES

To investigate how different competing interest (COI) statements affect clinical readers' perceptions of education articles.

DESIGN

Randomised controlled trial.

SETTING AND PARTICIPANTS

Random sample of UK doctors.

INTERVENTIONS

We created four permutations of each of two clinical reviews (on gout or dyspepsia), which varied only in terms of the COI statement. Volunteers were blinded and randomised to receive one review and asked to complete a questionnaire after reading it. Blinded factorial analyses of variance and analyses of covariance were carried out to assess the influence of each review and type of COI on outcomes.

PRIMARY AND SECONDARY OUTCOMES

Confidence in the article's conclusions (primary outcome), its importance, their level of interest in the article and their likelihood to change practice after reading it.

RESULTS

Of 10 889 doctors invited to participate, 1065 (10%) volunteered. Of these, 749 (70%) completed the survey. Analysis of covariance (adjusting for age, sex, job type, years since qualification) showed no significant difference between the groups in participants' confidence in the article (gout: p=0.32, dyspepsia: p=0.78) or their rating of its importance (gout: p=0.09, dyspepsia: p=0.79). For the gout review, participants rated articles with advisory board and consultancies COI as significantly less interesting than those with no COI (p=0.028 with Bonferroni correction). Among participants indicating that they treat the condition and that the article's recommendations differed from their own practice, there was no significant difference in likelihood to change practice between groups (gout: p=0.59, n=59; dyspepsia: p=0.56, n=80).

CONCLUSIONS

Doctors' confidence in educational articles was not influenced by the COI statements. Further work is required to determine if doctors do not perceive these COIs as important in educational articles or if they do not pay attention to these statements. More meaningful COI disclosure practices may be needed, which highlight context-specific potential sources of bias to readers.

TRIAL REGISTRATION NUMBER

NCT02548312; Results.

Making headlines: an analysis of US government-funded cancer research mentioned in online media

OBJECTIVE

To characterise how online media coverage of journal articles on cancer funded by the US government varies by cancer type and stage of the cancer control continuum and to compare the disease prevalence rates with the amount of funded research published for each cancer type and with the amount of media attention each receives.

DESIGN

A cross-sectional study.

SETTING

The United States.

PARTICIPANTS

The subject of analysis was 11 436 journal articles on cancer funded by the US government published in 2016. These articles were identified via PubMed and characterised as receiving online media attention based on data provided by Altmetric.

RESULTS

16.8% (n=1925) of articles published on US government-funded research were covered in the media. Published journal articles addressed all common cancers. Frequency of journal articles differed substantially across the common cancers, with breast cancer (n=1284), lung cancer (n=630) and prostate cancer (n=586) being the subject of the most journal articles. Roughly one-fifth to one-fourth of journal articles within each cancer category received online media attention. Media mentions were disproportionate to actual burden of each cancer type (ie, incidence and mortality), with breast cancer articles receiving the most media mentions. Scientific articles also covered the stages of the cancer continuum to varying degrees. Across the 13 most common cancer types, 4.4% (n=206) of articles focused on prevention and control, 11.7% (n=550) on diagnosis and 10.7% (n=502) on therapy.

CONCLUSIONS

Findings revealed a mismatch between prevalent cancers and cancers highlighted in online media. Further, journal articles on cancer control and prevention received less media attention than other cancer continuum stages. Media mentions were not proportional to actual public cancer burden nor volume of scientific publications in each cancer category. Results highlight a need for continued research on the role of media, especially online media, in research dissemination.

Fast-tracking acute stroke care in China: Shenzhen Stroke Emergency Map

China has the largest stroke population and at-risk population in the world. However, it has a lower thrombolytic therapy rate and longer onset-to-needle time/door-to-needle time for patients who had an acute stroke compared with developed countries, which might be due to redundant procedures or inefficient systems. Things are changing due to some new initiatives. Two years ago, a new emergency system in China, Stroke Emergency Map, was first launched as a regional emergency system in Shenzhen, the bustling metropolis just north of Hong Kong. As a result of the Stroke Emergency Map in Shenzhen, the number of thrombolytic cases increased in the last 2 years, from 568 to 809 annually. The Stroke Emergency Map, first pioneered in Shenzhen and now spreading to the rest of China, is a comprehensive and interdisciplinary system. The benefits are not just the immediate improvements in the acute stroke care because the continuous data collection and audit allows for improvements in logistics and future strategies.

Health Journalism: Health Reporting Status and Challenges

Background:

Media play crucial role in disseminating health information. Due to the importance of accurate health news reports, and the national need to professionalism in health journalism, this study aimed to investigate the characteristics of health journalists, and health reporting status and the challenges involved.

Materials and Methods:

Using consensus sampling, this descriptive cross-sectional study was conducted on all health news reporters in Isfahan (34 journalists) in 2015–2016. Data collection was done via a researcher-made questionnaire. Content validity of the questionnaire was determined by qualitative method and based on the opinions of six experts. The test–retest reliability coefficient was 98.0. Data analysis was done by Statistical Package for the Social Sciences, version 16 and descriptive statistics and content analysis were used for analyzing the responses to two open questions.

Results:

Among 34 journalists, 56% were women and 44% men; the majority of journalists (65%) had no specialized training on health reporting, 35% of journalists were not able to understand the health issues, and the knowledge of medical terminology in 59% of them was moderate to low. The most important required skill for reporters was the ability to interpret medical research reports (88%), 97% were eager to participate in specialized health education.

Conclusions:

Our study showed that health journalists lacked knowledge and specialized training for dissemination of health news. This has brought about serious challenges. Thus, development and implementation of training courses in close collaboration with educational department of the Ministry of Health and news programs professionals at Islamic Republic of Iran Broadcasting is highly recommended.

Promoting the Quality of Health Research-based News: Introduction of a Tool

Introduction:

While disseminating health research findings to the public, it is very important to present appropriate and accurate information to give the target audience a correct understanding of the subject matter. The objective of this study was to design and psychometrically evaluate a checklist for health journalists to help them prepare news of appropriate accuracy and authenticity.

Methods:

The study consisted of two phases, checklist design and psychometrics. Literature review and expert opinion were used to extract the items of the checklist in the first phase. In the second phase, to assess content and face validity, the judgment of 38 persons (epidemiologists with a tool production history, editors-in-chief, and health journalists) was used to check the items’ understandability, nonambiguity, relevancy, and clarity. Reliability was assessed by the test–retest method using intra-cluster correlation (ICC) indices in the two phases. Cronbach's alpha was used to assess internal validity of the checklist.

Results:

Based on the participants’ opinions, the items were reduced from 20 to 14 in number. The items were categorized into the following three domains: (a) items assessing the source of news and its validity, (b) items addressing the presentation of complete and accurate information on research findings, and (c) items which if adhered to lead to the target audiences’ better understanding. The checklist was approved for content and face validity. The reliability of the checklist was assessed in the last stage; the ICC was 1 for 12 items and above 0.8 for the other two. Internal consistency (Cronbach's alpha) was 0.98.

Discussion and Conclusions:

The resultant indices of the study indicate that the checklist has appropriate validity and reliability. Hence, it can be used by health journalists to develop health research-based news.

National citation patterns of NEJM, The Lancet, JAMA and The BMJ in the lay press: a quantitative content analysis

OBJECTIVES

To analyse the total number of newspaper articles citing the four leading general medical journals and to describe national citation patterns.

DESIGN

Quantitative content analysis.

SETTING/SAMPLE

Full text of 22 general newspapers in 14 countries over the period 2008-2015, collected from LexisNexis. The 14 countries have been categorised into four regions: the USA, the UK, Western World (European countries other than the UK, and Australia, New Zealand and Canada) and Rest of the World (other countries).

MAIN OUTCOME MEASURE

Press citations of four medical journals (two American: NEJM and JAMA; and two British: The Lancet and The BMJ) in 22 newspapers.

RESULTS

British and American newspapers cited some of the four analysed medical journals about three times a week in 2008-2015 (weekly mean 3.2 and 2.7 citations, respectively); the newspapers from other Western countries did so about once a week (weekly mean 1.1), and those from the Rest of the World cited them about once a month (monthly mean 1.1). The New York Times cited above all other newspapers (weekly mean 4.7). The analysis showed the existence of three national citation patterns in the daily press: American newspapers cited mostly American journals (70.0% of citations), British newspapers cited mostly British journals (86.5%) and the rest of the analysed press cited more British journals than American ones. The Lancet was the most cited journal in the press of almost all Western countries outside the USA and the UK. Multivariate correspondence analysis confirmed the national patterns and showed that over 85% of the citation data variability is retained in just one single new variable: the national dimension.

CONCLUSION

British and American newspapers are the ones that cite the four analysed medical journals more often, showing a domestic preference for their respective national journals; non-British and non-American newspapers show a common international citation pattern.

[How evidence-based are print- and online mass media in Austria? A quantitative analysis]

We rated the body of evidence for 219 health-related questions that had been covered by 990 media articles in Austrian print and online media. In 59.5 % of these articles, the evidence for medical facts is reported in a highly distorted (exaggerated or understated) manner; only 10.8 % adequately reflect the actual strength of evidence. While 61.3% of the articles imply that the reported effects are based on the highest level of evidence, it really only applies to 2.6% of the articles. Compared to quality media, tabloid media report in a more distorted way. However, this is mainly due to a different subject coverage, since the degree of distorted reporting does not significantly differ for subjects covered by both tabloid and quality media. Online media do not report in a more distorted way than print media, with the exception of the newspaper "Der Standard". A clear difference can be seen in the reporting on different subjects. Articles on medications regulated by governmental authorities or interventions which only physicians are allowed to perform were significantly less distorted than articles on nutritional supplements or interventions that may be provided by non-physician staff. Reports on cosmetic or weight loss interventions were most distorted.

Clinical study reports of randomised controlled trials: an exploratory review of previously confidential industry reports

OBJECTIVE

To explore the structure and content of a non-random sample of clinical study reports (CSRs) to guide clinicians and systematic reviewers.

SEARCH STRATEGY

We searched public sources and lodged Freedom of Information requests for previously confidential CSRs primarily written by the industry for regulators.

SELECTION CRITERIA

CSRs reporting sufficient information for extraction ('adequate').

PRIMARY OUTCOME MEASURES

Presence and length of essential elements of trial design and reporting and compression factor (ratio of page length for CSRs compared to its published counterpart in a scientific journal).

DATA EXTRACTION

Data were extracted on standard forms and crosschecked for accuracy.

RESULTS

We assembled a population of 78 CSRs (covering 90 randomised controlled trials; 144 610 pages total) dated 1991-2011 of 14 pharmaceuticals. Report synopses had a median length of 5 pages, efficacy evaluation 13.5 pages, safety evaluation 17 pages, attached tables 337 pages, trial protocol 62 pages, statistical analysis plan 15 pages and individual efficacy and safety listings had a median length of 447 and 109.5 pages, respectively. While 16 (21%) of CSRs contained completed case report forms, these were accessible to us in only one case (765 pages representing 16 individuals). Compression factors ranged between 1 and 8805.

CONCLUSIONS

Clinical study reports represent a hitherto mostly hidden and untapped source of detailed and exhaustive data on each trial. They should be consulted by independent parties interested in a detailed record of a clinical trial, and should form the basic unit for evidence synthesis as their use is likely to minimise the problem of reporting bias. We cannot say whether our sample is representative and whether our conclusions are generalisable to an undefined and undefinable population of CSRs.

Fate of manuscripts rejected by a non-English-language general medical journal: a retrospective cohort study

OBJECTIVE

The objective of this study was to determine whether, where and when manuscripts were published following rejection by the Journal of the Danish Medical Association, a general medical journal published in Danish. Similar previous studies have focused on specialty/subspecialty journals published in English.

DESIGN

Manuscripts rejected during a 4-year period were searched for in PubMed and Embase in order to assess the percentage of manuscripts subsequently published in other journals. In addition, characteristics of both the published manuscripts and the journals in which they were evaluated.

RESULTS

Of 198 rejected manuscripts, 21 (10.6%) were eventually published after a median of 685 days (range 209-1463). The majority of these were original research, published in English-language specialty/subspecialty journals. The median number of citations per article was 2-3 (IQR 0.5-9.5, depending on the database searched).

CONCLUSIONS

10.6% of the rejected manuscripts were eventually published in other journals, mainly English-language specialty journals. This proportion was considerably lower than that for other journals that have studied the fate of rejected manuscripts. Manuscript translation could be a barrier for resubmitting to English-language journals with larger readerships, thus hindering the dissemination of knowledge to the international community.

Could media reports include a standardized scale for quality of evidence?

While some media reports offer accurate interpretations of clinical research, other reports are misleading. The uneven accuracy of medical reporting may act in concert with its sheer volume to confuse the lay public about which health messages are most important and evidence-based. I outline one possible step towards a solution: medical journals can embed quality of evidence ratings in article summaries and create incentives for inclusion of these ratings in lay media reports.