Languages for different health information readers: multitrait-multimethod content analysis of Cochrane systematic reviews textual summary formats

Conclusiveness, readability and textual characteristics of plain language summaries from medical and non-medical organizations: a cross-sectional study

This cross-sectional study compared plain language summaries (PLSs) from medical and non-medical organizations regarding conclusiveness, readability and textual characteristics. All Cochrane (medical PLSs, n = 8638) and Campbell Collaboration and International Initiative for Impact Evaluation (non-medical PLSs, n = 163) PLSs of latest versions of systematic reviews published until 10 November 2022 were analysed. PLSs were classified into three conclusiveness categories (conclusive, inconclusive and unclear) using a machine learning tool for medical PLSs and by two experts for non-medical PLSs. A higher proportion of non-medical PLSs were conclusive (17.79% vs 8.40%, P < 0.0001), they had higher readability (median number of years of education needed to read the text with ease 15.23 (interquartile range (IQR) 14.35 to 15.96) vs 15.51 (IQR 14.31 to 16.77), P = 0.010), used more words (median 603 (IQR 539.50 to 658.50) vs 345 (IQR 202 to 476), P < 0.001). Language analysis showed that medical PLSs scored higher for disgust and fear, and non-medical PLSs scored higher for positive emotions. The reason for the observed differences between medical and non-medical fields may be attributed to the differences in publication methodologies or disciplinary differences. This approach to analysing PLSs is crucial for enhancing the overall quality of PLSs and knowledge translation to the general public.

Retrieval augmentation of large language models for lay language generation

The complex linguistic structures and specialized terminology of expert-authored content limit the accessibility of biomedical literature to the general public. Automated methods have the potential to render this literature more interpretable to readers with different educational backgrounds. Prior work has framed such lay language generation as a summarization or simplification task. However, adapting biomedical text for the lay public includes the additional and distinct task of background explanation: adding external content in the form of definitions, motivation, or examples to enhance comprehensibility. This task is especially challenging because the source document may not include the required background knowledge. Furthermore, background explanation capabilities have yet to be formally evaluated, and little is known about how best to enhance them. To address this problem, we introduce Retrieval-Augmented Lay Language (RALL) generation, which intuitively fits the need for external knowledge beyond that in expert-authored source documents. In addition, we introduce CELLS, the largest (63k pairs) and broadest-ranging (12 journals) parallel corpus for lay language generation. To evaluate RALL, we augmented state-of-the-art text generation models with information retrieval of either term definitions from the UMLS and Wikipedia, or embeddings of explanations from Wikipedia documents. Of these, embedding-based RALL models improved summary quality and simplicity while maintaining factual correctness, suggesting that Wikipedia is a helpful source for background explanation in this context. We also evaluated the ability of both an open-source Large Language Model (Llama 2) and a closed-source Large Language Model (GPT-4) in background explanation, with and without retrieval augmentation. Results indicate that these LLMs can generate simplified content, but that the summary quality is not ideal. Taken together, this work presents the first comprehensive study of background explanation for lay language generation, paving the path for disseminating scientific knowledge to a broader audience. Our code and data are publicly available at: https://github.com/LinguisticAnomalies/pls_retrieval.

Investigating the Readability and Linguistic, Psychological, and Emotional Characteristics of Digital Dementia Information Written in the English Language: Multitrait-Multimethod Text Analysis

BACKGROUND

Past research in the Western context found that people with dementia search for digital dementia information in peer-reviewed medical research articles, dementia advocacy and medical organizations, and blogs written by other people with dementia. This past work also demonstrated that people with dementia do not perceive English digital dementia information as emotionally or cognitively accessible.

OBJECTIVE

In this study, we sought to investigate the readability; linguistic, psychological, and emotional characteristics; and target audiences of digital dementia information. We conducted a textual analysis of 3 different types of text-based digital dementia information written in English: 300 medical articles, 35 websites, and 50 blogs.

METHODS

We assessed the text's readability using the Flesch Reading Ease and Flesch-Kincaid Grade Level measurements, as well as tone, analytical thinking, clout, authenticity, and word frequencies using a natural language processing tool, Linguistic Inquiry and Word Count Generator. We also conducted a thematic analysis to categorize the target audiences for each information source and used these categorizations for further statistical analysis.

RESULTS

The median Flesch-Kincaid Grade Level readability score and Flesch Reading Ease score for all types of information (N=1139) were 12.1 and 38.6, respectively, revealing that the readability scores of all 3 information types were higher than the minimum requirement. We found that medical articles had significantly (P=.05) higher word count and analytical thinking scores as well as significantly lower clout, authenticity, and emotional tone scores than websites and blogs. Further, blogs had significantly (P=.48) higher word count and authenticity scores but lower analytical scores than websites. Using thematic analysis, we found that most of the blogs (156/227, 68.7%) and web pages (399/612, 65.2%) were targeted at people with dementia. Website information targeted at a general audience had significantly lower readability scores. In addition, website information targeted at people with dementia had higher word count and lower emotional tone ratings. The information on websites targeted at caregivers had significantly higher clout and lower authenticity scores.

CONCLUSIONS

Our findings indicate that there is an abundance of digital dementia information written in English that is targeted at people with dementia, but this information is not readable by a general audience. This is problematic considering that people with <12 years of education are at a higher risk of developing dementia. Further, our findings demonstrate that digital dementia information written in English has a negative tone, which may be a contributing factor to the mental health crisis many people with dementia face after receiving a diagnosis. Therefore, we call for content creators to lower readability scores to make the information more accessible to a general audience and to focus their efforts on providing information in a way that does not perpetuate overly negative narratives of dementia.

Linguistic analysis of plain language summaries and corresponding scientific summaries of Cochrane systematic reviews about oncology interventions

BACKGROUND

Cochrane plain language summaries (PLSs) are an important format to present high-quality healthcare evidence to patients with cancer and their families. They should be written in a way everyone can understand, since they serve as a tool in decision-making and present a bridge to overcome the gap between the healthcare users and professionals.

OBJECTIVE

The aim of the study was to assess the language characteristics of PLSs of Cochrane systematic reviews of oncology interventions in comparison with corresponding Cochrane scientific abstracts (SAs).

METHODS

In this cross-sectional study, we included all Cochrane PLSs and SAs of systematic reviews of oncology interventions available in the Cochrane Database of Systematic Reviews. We assessed text readability, measured using the Simple Measure of Gobbledygook (SMOG) index, and the prevalence of words related to different language tones (clout, authenticity, emotions and analytical tones). Two independent assessors categorized the conclusiveness of the efficacy of interventions into nine categories.

RESULTS

The overall median SMOG index for 275 PLSs was 13.0 (95% confidence interval [CI] 12.8-13.3). Readability scores did not differ across Cochrane Review Groups. SAs had a higher readability index than the corresponding PLSs (median = 16.6, 95% CI = 16.4-16.8). Regarding linguistic characteristics, PLSs were shorter than SAs, with less use of analytical tone, but more use of a positive emotional tone and authenticity. Overall, the 'Unclear' category of conclusiveness was the most common among all PLSs. Also, PLSs with 'No evidence' conclusions were the shortest and had the lowest SMOG index.

CONCLUSION

PLSs of Cochrane systematic reviews of oncological interventions have low readability and most give unclear conclusions about the efficacy of interventions. PLSs should be simplified so that patients and their families can benefit from appropriate health information on evidence synthesis. Further research is needed into reasons for unclear language to describe evidence from oncology trials.

Demands on Health Information and Clinical Practice Guidelines for Patients from the Perspective of Adults with Mental Illness and Family Members: A Qualitative Study with In-Depth Interviews

(1) Background: "Patient health information" promote health literacy. "Patient guidelines" as a sub group reflect the current evidence about illnesses and treatment options adapted to the needs of laypersons. Little is known about factors promoting and hindering their use by people affected by mental illness and their relatives. (2) Methods: Telephone interviews (N = 15; n = 4 adults affected by mental illness, n = 5 relatives, n = 6 both applicable) were conducted according to the Sørensen model of health literacy. Data were recorded, transcribed and content-analyzed following Mayring. (3) Results: Health information is used regularly by individuals affected by mental illness and their relatives, but "patient guidelines" are largely unknown. Yet, there is a great willingness to use them. Main barriers are a lack of statistical knowledge, the complexity of health-related topics and cognitive impairment sometimes accompanying mental illnesses. Target group-oriented adaptation as well as transparent and even-handed presentation of (dis-)advantages of treatment options can increase trust. (4) Conclusions: Health information and guidelines can help affected persons and relatives to make treatment decisions by conveying unbiased, up-to-date knowledge. Target group-specific adaptations should be made for psychiatric illnesses and features specific to mental illnesses compared to physical illnesses should be included. Clinical practice guidelines must be distributed more widely to increase their impact.

The effectiveness and acceptability of evidence synthesis summary formats for clinical guideline development groups: a mixed-methods systematic review

INTRODUCTION

Clinical guideline development often involves a rigorous synthesis of evidence involving multidisciplinary stakeholders with different priorities and knowledge of evidence synthesis; this makes communicating findings complex. Summary formats are typically used to communicate the results of evidence syntheses; however, there is little consensus on which formats are most effective and acceptable for different stakeholders.

METHODS

This mixed-methods systematic review (MMSR) aimed to evaluate the effectiveness and acceptability (e.g. preferences and attitudes and preferences towards) of evidence synthesis summary formats for GDG members. We followed the PRISMA 2020 guideline and Joanna Briggs Institute Manual for Evidence Synthesis for MMSRs. We searched six databases (inception to April 20, 2021) for randomised controlled trials (RCTs), RCTs with a qualitative component, and qualitative studies. Screening, data extraction, and quality appraisal were performed in duplicate. Qualitative findings were synthesised using meta-aggregation, and quantitative findings are described narratively.

RESULTS

We identified 17,240 citations and screened 54 full-text articles, resulting in 22 eligible articles (20 unique studies): 4 articles reported the results of 5 RCTs, one of which also had a qualitative component. The other 18 articles discussed the results of 16 qualitative studies. Therefore, we had 5 trials and 17 qualitative studies to extract data from. Studies were geographically heterogeneous and included a variety of stakeholders and summary formats. All 5 RCTs assessed knowledge or understanding with 3 reporting improvement with newer formats. The qualitative analysis identified 6 categories of recommendations: 'presenting information', 'tailoring information' for end users, 'trust in producers and summary', 'knowledge required' to understand findings, 'quality of evidence', and properly 'contextualising information'. Across these categories, the synthesis resulted in 126 recommendations for practice. Nine recommendations were supported by both quantitative and qualitative evidence and 116 by only qualitative. A majority focused on how to present information (n = 64) and tailor content for different end users (n = 24).

CONCLUSIONS

This MMSR provides guidance on how to improve evidence summary structure and layout. This can be used by synthesis producers to better communicate to GDGs. Study findings will inform the co-creation of evidence summary format prototypes based on GDG member's needs. Trial registration The protocol for this project was previously published, and the project was preregistered on Open Science Framework (Clyne and Sharp, Evidence synthesis and translation of findings for national clinical guideline development: addressing the needs and preferences of guideline development groups, 2021; Sharp and Clyne, Evidence synthesis summary formats for decision-makers and Clinical Guideline Development Groups: A mixed-methods systematic review protocol, 2021).

Effectiveness of letters to patients with or without Cochrane blogshots on 10-year cardiovascular risk change among women in menopausal transition: 6-month three-arm randomized controlled trial

BACKGROUND

Health information and patient education on lifestyle changes may have a positive effect on the prevention of many chronic conditions, especially cardiovascular diseases (CVDs). We performed a parallel, three-arm randomized controlled trial (RCT) of 6-month educational intervention in a form of letters containing a reminder of the participant's CVD risk with or without Cochrane blogshots to reduce CVD risk among women aged 45-65 with one or more known CVD risk factors.

METHODS

The control group received a letter about their CVD risk at the beginning of the trial. The intervention groups received the initial letter about their CVD risk and remainder letters about their CVD risk every 2 months, with or without Cochrane blogshots: (1) effect of calcium in the prevention of high blood pressure, (2) effect of reducing saturated fat acids in eating habits, and (3) effects of green and black tea in CVD prevention. The primary outcome was CVD risk reduction calculated as the difference between the baseline and 6-month score for a 10-year risk of fatal CVD according to the ACC/AHA guidelines.

RESULTS

After both interventions, CVD risk reduction was significantly higher compared to the control group (P < 0.001, Kruskal-Wallis H test). The number of participants who decreased their CV risk was 29% (20/70) in the control group, 69% (48/70) in the group receiving the reminder letters, and 70% (49/70) in the group receiving the reminder letters and blogshots. The number needed to treat to achieve risk reduction was 2.41 (95% CI = 1.77 to 3.78) for letters with a CVD risk reminder and 2.50 (1.81 to 4.03) for letters with a reminder and a blogshot. The group receiving reminder letters with Cochrane blogshots had a significant change in the category of CVD risk, mainly from high to moderate and from moderate to low CVD risk category.

CONCLUSIONS

A simple and inexpensive intervention method in a form of letters reminding women about their CVD risk with or without providing additional health information in the form of Cochrane blogshots about interventions for important CVD risk factors may be effective in CVD management and could be considered by primary care providers.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT04601558. Retrospectively registered on October 19, 2020.

Conclusiveness, linguistic characteristics and readability of Cochrane plain language summaries of intervention reviews: a cross-sectional study

Background

One of the most important formats to disseminate the evidence in health to different populations are Cochrane Plain Language Summaries (PLSs). PLSs should be written in a simplified language, easily understandable and providing clear message for the consumer. The aim of this study was to examine the extent to which PLSs are customized for lay persons, specifically by providing conclusive, comprehensible, and readable messages.

Methods

The study analyzed Cochrane PLSs of interventional studies (N = 4360) in the English language published from 1995 to 2019. We categorized the conclusiveness into one of the following categories: “positive”, “positive inconclusive”, “no evidence”, “no opinion”, “negative”, “negative inconclusive”, “unclear”, “equal”, “equal inconclusive”. Language characteristics were analyzed using Linguistic Inquiry and Word Count (LIWC) software. The level of readability was measured by SMOG (Simple Measure of Gobbledygook) index, indicating the number of years of education required to read the text. For each PLS, we also collected the following data: Cochrane Review Network, year of publication and number of authors.

Results

Most of the PLSs (80%) did not have a conclusive message. In 53% PLSs there was no concluding opinion about the studied intervention or the conclusion was unclear. The most frequent conclusiveness category was “no opinion” (30%), and its frequency increased over time. The conclusiveness categories were similarly dispersed across Cochrane Networks. PLSs were written in an objective style, with high levels of analytical tone and clout above neutral, but a lower relation to authenticity and tone. The median number of years of non-specific education needed to read the PLSs was 14.9 (IQR 13.8 to 16.1), indicating that the person needs almost 15 years of general education to read the content with ease.

Conclusion

Most of the Cochrane PLSs provided no concluding opinion or unclear conclusion regarding the effects of analyzed intervention. Analysis of readability indicated that they may be difficult to read for the lay population without medical education. Our results indicate that PLSs may not be so plain, and that the writing of Cochrane PLSs requires more effort. Tools used in this study could improve PLSs and make them better suited for lay audiences.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01721-7.

Plain language summaries: A systematic review of theory, guidelines and empirical research

Plain language summaries (PLSs) have been introduced to communicate research in an understandable way to a nonexpert audience. Guidelines for writing PLSs have been developed and empirical research on PLSs has been conducted, but terminology and research approaches in this comparatively young field vary considerably. This prompted us to review the current state of the art of the theoretical and empirical literature on PLSs. The two main objectives of this review were to develop a conceptual framework for PLS theory, and to synthesize empirical evidence on PLS criteria. We began by searching Web of Science, PubMed, PsycInfo and PSYNDEX (last search 07/2021). In our review, we included empirical investigations of PLSs, reports on PLS development, PLS guidelines, and theoretical articles referring to PLSs. A conceptual framework was developed through content analysis. Empirical studies investigating effects of PLS criteria on defined outcomes were narratively synthesized. We identified 7,714 records, of which 90 articles met the inclusion criteria. All articles were used to develop a conceptual framework for PLSs which comprises 12 categories: six of PLS aims and six of PLS characteristics. Thirty-three articles empirically investigated effects of PLSs on several outcomes, but study designs were too heterogeneous to identify definite criteria for high-quality PLSs. Few studies identified effects of various criteria on accessibility, understanding, knowledge, communication of research, and empowerment. We did not find empirical evidence to support most of the criteria we identified in the PLS writing guidelines. We conclude that although considerable work on establishing and investigating PLSs is available, empirical evidence on criteria for high-quality PLSs remains scarce. The conceptual framework developed in this review may provide a valuable starting point for future guideline developers and PLS researchers.

Optimizing Readability and Format of Plain Language Summaries for Medical Research Articles: Cross-sectional Survey Study

Background

Plain language summaries (PLSs) are intended to provide readers with a clear, nontechnical, and easily understandable overview of medical and scientific literature; however, audience preferences for specific PLS formats have yet to be fully explored.

Objective

This study aims to evaluate the preferred readability level and format for PLSs of medical research articles of different disease states via a web-based survey of audiences of different age groups.

Methods

Articles describing phase III clinical trials published in top-level, peer-reviewed journals between May 2016 and May 2018 were identified for 3 chronic disease states representing a range of adult patient age groups: (1) psoriasis, a skin disease representative of younger patients; (2) multiple sclerosis (MS), a neurological disease representative of middle-aged patients; and (3) rheumatoid arthritis (RA), a painful joint disease representative of older patients. Four PLSs were developed for each research article, of which 3 were text-only summaries (written with high, medium, and low complexity) and 1 was an infographic. To evaluate each of the 4 PLS formats, a 20-question open survey (specific to one of the 3 diseases) was sent to a representative sample selected via UK-based patient association websites, Twitter, and Facebook patient groups. A weighted-average calculation was applied to respondents’ ranked preferences for each PLS format.

Results

For all 3 articles, the weighted-average preference scores showed that infographic (psoriasis 2.91, MS 2.71, and RA 2.78) and medium-complexity text-based PLS (reading age 14-17 years, US Grade 9-11; psoriasis 2.90; MS 2.47; RA 2.77) were the two most preferred PLS formats.

Conclusions

Audience preferences should be accounted for when preparing PLSs to accompany peer-reviewed original research articles. Oversimplified text can be viewed negatively, and graphical summaries or medium-complexity text-based summaries appear to be the most popular.

Plain Language Summary

Patients and caregivers should have the chance to read about medical research in a format they can understand. However, we do not know much about the formats that people with different illnesses or ages prefer. Researchers wanted to find out more about this. They selected 3 medical articles about illnesses that affect different age groups: psoriasis (younger patients), multiple sclerosis (middle-aged patients), and rheumatoid arthritis (older patients). They created 4 summaries of each article. One was a graphical summary, and the other 3 were words-only summaries of high, medium, and low complexity. Then, the researchers posted surveys on UK patient group websites and Facebook patient groups to ask people what they thought of the summaries. The surveys were taken by 167 people. These people were patients with psoriasis, multiple sclerosis, or rheumatoid arthritis, or their caregivers. Most were women, and about half had a university degree. For each illness, most people preferred the graphical summary. Among the word-only summaries, most people preferred the medium-complexity wording written for a reading age of 14 to 17 years. People felt that the graphical and medium-complexity summaries were clear and concise, while the others used jargon or were too simple. Authors of medical articles should remember these results when writing summaries for patients. More research is needed about the preferences of other people, such as those with other illnesses. (See Multimedia Appendix 1 for the graphical summary of the plain language summary.)

Sentiment Analysis for Necessary Preview of 30-Day Mortality in Sepsis Patients and the Control Strategies

This study was to preview the risk of 30-day mortality in sepsis patients using sentiment analysis. The clinical data of patients and nursing notes were collected from the Medical Information Mart for Intensive Care (MIMIC-III) database. The factors influencing 30-day mortality were analyzed using the Cox regression model. And, the prognostic index (PI) was estimated. The receiver operating characteristic (ROC) curve was used to determine the PI cut-off point and assess the prediction ability of the model. In total, 1844 of 3560 patients were eligible for the study, with a 30-day mortality of 37.58%. Multivariate Cox analysis showed that sentiment polarity scores, sentiment subjectivity scores, simplified acute physiology score (SAPS)-II, age, and intensive care unit (ICU) types were all associated with the risk of 30-day mortality (P < 0.05). In the preview of 30-day mortality, the area under the curve (AUC) of ROC was 0.78 (95%CI: 0.74–0.81,P < 0.001) when the cut-off point of PI was 0.467. The documented notes from nurses were described for the first time. Sentiment scores measured in nursing notes are associated with the risk of 30-day mortality in sepsis patients and may improve the preview of 30-day mortality.

Predicting the Easiness and Complexity of English Health Materials for International Tertiary Students With Linguistically Enhanced Machine Learning Algorithms: Development and Validation Study

BACKGROUND

There is an increasing body of research on the development of machine learning algorithms in the evaluation of online health educational resources for specific readerships. Machine learning algorithms are known for their lack of interpretability compared with statistics. Given their high predictive precision, improving the interpretability of these algorithms can help increase their applicability and replicability in health educational research and applied linguistics, as well as in the development and review of new health education resources for effective and accessible health education.

OBJECTIVE

Our study aimed to develop a linguistically enriched machine learning model to predict binary outcomes of online English health educational resources in terms of their easiness and complexity for international tertiary students.

METHODS

Logistic regression emerged as the best performing algorithm compared with support vector machine (SVM) (linear), SVM (radial basis function), random forest, and extreme gradient boosting on the transformed data set using L2 normalization. We applied recursive feature elimination with SVM to perform automatic feature selection. The automatically selected features (n=67) were then further streamlined through expert review. The finalized feature set of 22 semantic features achieved a similar area under the curve, sensitivity, specificity, and accuracy compared with the initial (n=115) and automatically selected feature sets (n=67). Logistic regression with the linguistically enhanced feature set (n=22) exhibited important stability and robustness on the training data of different sizes (20%, 40%, 60%, and 80%), and showed consistently high performance when compared with the other 4 algorithms (SVM [linear], SVM [radial basis function], random forest, and extreme gradient boosting).

RESULTS

We identified semantic features (with positive regression coefficients) contributing to the prediction of easy-to-understand online health texts and semantic features (with negative regression coefficients) contributing to the prediction of hard-to-understand health materials for readers with nonnative English backgrounds. Language complexity was explained by lexical difficulty (rarity and medical terminology), verbs typical of medical discourse, and syntactic complexity. Language easiness of online health materials was associated with features such as common speech act verbs, personal pronouns, and familiar reasoning verbs. Successive permutation of features illustrated the interaction between these features and their impact on key performance indicators of the machine learning algorithms.

CONCLUSIONS

The new logistic regression model developed exhibited consistency, scalability, and, more importantly, interpretability based on existing health and linguistic research. It was found that low and high linguistic accessibilities of online health materials were explained by 2 sets of distinct semantic features. This revealed the inherent complexity of effective health communication beyond current readability analyses, which were limited to syntactic complexity and lexical difficulty.

Paragraph-level Simplification of Medical Texts

We consider the problem of learning to simplify medical texts. This is important because most reliable, up-to-date information in biomedicine is dense with jargon and thus practically inaccessible to the lay audience. Furthermore, manual simplification does not scale to the rapidly growing body of biomedical literature, motivating the need for automated approaches. Unfortunately, there are no large-scale resources available for this task. In this work we introduce a new corpus of parallel texts in English comprising technical and lay summaries of all published evidence pertaining to different clinical topics. We then propose a new metric based on likelihood scores from a masked language model pretrained on scientific texts. We show that this automated measure better differentiates between technical and lay summaries than existing heuristics. We introduce and evaluate baseline encoder-decoder Transformer models for simplification and propose a novel augmentation to these in which we explicitly penalize the decoder for producing 'jargon' terms; we find that this yields improvements over baselines in terms of readability.

Use of Machine Learning Algorithms to Predict the Understandability of Health Education Materials: Development and Evaluation Study

BACKGROUND

Improving the understandability of health information can significantly increase the cost-effectiveness and efficiency of health education programs for vulnerable populations. There is a pressing need to develop clinically informed computerized tools to enable rapid, reliable assessment of the linguistic understandability of specialized health and medical education resources. This paper fills a critical gap in current patient-oriented health resource development, which requires reliable and accurate evaluation instruments to increase the efficiency and cost-effectiveness of health education resource evaluation.

OBJECTIVE

We aimed to translate internationally endorsed clinical guidelines to machine learning algorithms to facilitate the evaluation of the understandability of health resources for international students at Australian universities.

METHODS

Based on international patient health resource assessment guidelines, we developed machine learning algorithms to predict the linguistic understandability of health texts for Australian college students (aged 25-30 years) from non-English speaking backgrounds. We compared extreme gradient boosting, random forest, neural networks, and C5.0 decision tree for automated health information understandability evaluation. The 5 machine learning models achieved statistically better results compared to the baseline logistic regression model. We also evaluated the impact of each linguistic feature on the performance of each of the 5 models.

RESULTS

We found that information evidentness, relevance to educational purposes, and logical sequence were consistently more important than numeracy skills and medical knowledge when assessing the linguistic understandability of health education resources for international tertiary students with adequate English skills (International English Language Testing System mean score 6.5) and high health literacy (mean 16.5 in the Short Assessment of Health Literacy-English test). Our results challenge the traditional views that lack of medical knowledge and numerical skills constituted the barriers to the understanding of health educational materials.

CONCLUSIONS

Machine learning algorithms were developed to predict health information understandability for international college students aged 25-30 years. Thirteen natural language features and 5 evaluation dimensions were identified and compared in terms of their impact on the performance of the models. Health information understandability varies according to the demographic profiles of the target readers, and for international tertiary students, improving health information evidentness, relevance, and logic is critical.

Efficacy of reminders for increasing volunteer engagement in translating Cochrane plain language summaries: a pilot randomised controlled trial

OBJECTIVES

The aim of this study was to pilot test the effectiveness of reminders versus no intervention for increasing the number of translated Cochrane plain language summaries (PLSs) among volunteer translators.

STUDY DESIGN

Parallel-group randomised controlled trial.

SETTING

Cochrane Croatia translation project.

PARTICIPANTS

Adults who volunteered to translate Cochrane PLSs within the Cochrane Croatia translation project.

INTERVENTION

The participants were randomly allocated to intervention (receiving up to four bi-weekly email reminders to translate PLSs) or control group (no intervention).

PRIMARY OUTCOME

The number of translated PLSs within the 6-month trial period.

RESULTS

We included 80 participants. The median number of translated PLSs after 6 months was 9 in the intervention group (95% CI 2.0 to 15.0) and 4 in the control group (95% CI 2.9 to 7.0), but this was not significantly different (p=0.181, Mann-Whitney U test). There was no difference between the groups in the number of translations after 3 months, the average time-to-translation after 3 or 6 months, or the satisfaction at the end of the study period. The number of reminders received and the number of translated summaries were negatively correlated (r=-0.50; 95% CI -0.70 to -0.22).

CONCLUSIONS

Our pilot trial showed that reminders do not seem to be significantly effective in increasing the number of PLS translations. Future studies could explore whether different frequency, timing and content of reminders have an influence on an increase in the engagement among volunteer translators of evidence synthesis.

TRIAL REGISTRATION

NCT03534791.

Emotional tones in scientific writing: comparison of commercially funded studies and non-commercially funded orthopedic studies

Background and purpose - There is ongoing debate as to whether commercial funding influences reporting of medical studies. We asked: Is there a difference in reported tones between abstracts, introductions, and discussions of orthopedic journal studies that were commercially funded and those that were not commercially funded?Methods - We conducted a systematic PubMed search to identify commercially funded studies published in 20 orthopedic journals between January 1, 2000 and December 1, 2019. We identified commercial funding of studies by including in our search the names of 10 medical device companies with the largest revenue in 2019. Commercial funding was designated when either the study or 1 or more of the authors received funding from a medical device company directly related to the content of the study. We matched 138 commercially funded articles 1 to 1 with 138 non-commercially funded articles with the same study design, published in the same journal, within a time range of 5 years. The IBM Watson Tone Analyzer was used to determine emotional tones (anger, fear, joy, and sadness) and language style (analytical, confident, and tentative).Results - For abstract and introduction sections, we found no differences in reported tones between commercially funded and non-commercially funded studies. Fear tones (non-commercially funded studies 5.1%, commercially funded studies 0.7%, p = 0.04), and analytical tones (non-commercially funded studies 95%, commercially funded studies 88%, p = 0.03) were more common in discussions of studies that were not commercially funded.Interpretation - Commercially funded studies have comparable tones to non-commercially funded studies in the abstract and introduction. In contrast, the discussion of non-commercially funded studies demonstrated more fear and analytical tones, suggesting them to be more tentative, accepting of uncertainty, and dispassionate. As text analysis tools become more sophisticated and mainstream, it might help to discern commercial bias in scientific reports.

Large-scale language analysis of peer review reports

Peer review is often criticized for being flawed, subjective and biased, but research into peer review has been hindered by a lack of access to peer review reports. Here we report the results of a study in which text-analysis software was used to determine the linguistic characteristics of 472,449 peer review reports. A range of characteristics (including analytical tone, authenticity, clout, three measures of sentiment, and morality) were studied as a function of reviewer recommendation, area of research, type of peer review and reviewer gender. We found that reviewer recommendation had the biggest impact on the linguistic characteristics of reports, and that area of research, type of peer review and reviewer gender had little or no impact. The lack of influence of research area, type of review or reviewer gender on the linguistic characteristics is a sign of the robustness of peer review.

Comparison of blogshots with plain language summaries of Cochrane systematic reviews: a qualitative study and randomized trial

Background

Cochrane, an organization dedicated to the production and dissemination of high-quality evidence on health, endeavors to reach consumers by developing appropriate summary formats of its systematic reviews. However, the optimal type of presentation of evidence to consumers is still unknown.

Objective

The aim of this study was to investigate consumer preferences for different summary formats of Cochrane systematic reviews (CSRs), using both qualitative and quantitative approaches.

Methods

Initially, we conducted three focus groups with medical students (n = 7), doctors (n = 4), and patients (n = 9) in 2017 to explore their health information search habits and preferences for CSR summary formats. Based on those findings, we conducted a randomized trial with medical students at the University of Split School of Medicine, Croatia, and with patients from three Dalmatian family practices to determine whether they prefer CSR blogshots (n = 115) or CSR plain language summaries (PLSs; n = 123).

Results

Participants in the focus groups favored brief and explicit CSR summary formats with fewer numbers. Although we found no difference in participants’ preferences for a specific summary format in the overall sample, subgroup analysis showed that patients preferred blogshots over PLSs in comparison to medical students (P = 0.003, eta squared effect size η² = 0.04).

Conclusion

CSR summaries should be produced in a format that meets the expectations and needs of consumers. Use of blogshots as a summary format could enhance the dissemination of CSRs among patients.

Trial registration

The trial was registered in ClinicalTrials.gov, NCT03542201. Registered on May 31st 2018.

Scientific abstracts and plain language summaries in psychology: A comparison based on readability indices

Findings from psychological research are usually difficult to interpret for non-experts. Yet, non-experts resort to psychological findings to inform their decisions (e.g., whether to seek a psychotherapeutic treatment or not). Thus, the communication of psychological research to non-expert audiences has received increasing attention over the last years. Plain language summaries (PLS) are abstracts of peer-reviewed journal articles that aim to explain the rationale, methods, findings, and interpretation of a scientific study to non-expert audiences using non-technical language. Unlike media articles or other forms of accessible research summaries, PLS are usually written by the authors of the respective journal article, ensuring that research content is accurately reproduced. In this study, we compared the readability of PLS and corresponding scientific abstracts in a sample of 103 journal articles from two psychological peer-reviewed journals. To assess readability, we calculated four readability indices that quantify text characteristics related to reading comprehension (e.g., word difficulty, sentence length). Analyses of variance revealed that PLS were easier to read than scientific abstracts. This effect emerged in both included journals and across all readability indices. There was only little evidence that this effect differed in magnitude between the included journals. In sum, this study shows that PLS may be an effective instrument for communicating psychological research to non-expert audiences. We discuss future research avenues to increase the quality of PLS and strengthen their role in science communication.