Bad News: Analysis of the Quality of Information on Influenza Prevention Returned by Google in English and Italian

Evaluating online health information quality using machine learning and deep learning: A systematic literature review

Background

Due to the large volume of online health information, while quality remains dubious, understanding the usage of artificial intelligence to evaluate health information and surpass human-level performance is crucial. However, the existing studies still need a comprehensive review highlighting the vital machine, and Deep learning techniques for the automatic health information evaluation process.

Objective

Therefore, this study outlines the most recent developments and the current state of the art regarding evaluating the quality of online health information on web pages and specifies the direction of future research.

Methods

In this article, a systematic literature is conducted according to the PRISMA statement in eight online databases PubMed, Science Direct, Scopus, ACM, Springer Link, Wiley Online Library, Emerald Insight, and Web of Science to identify all empirical studies that use machine and deep learning models for evaluating the online health information quality. Furthermore, the selected techniques are compared based on their characteristics, such as health quality criteria, quality measurement tools, algorithm type, and achieved performance.

Results

The included papers evaluate health information on web pages using over 100 quality criteria. The results show no universal quality dimensions used by health professionals and machine or deep learning practitioners while evaluating health information quality. In addition, the metrics used to assess the model performance are not the same as those used to evaluate human performance.

Conclusions

This systemic review offers a novel perspective in approaching the health information quality in web pages that can be used by machine and deep learning practitioners to tackle the problem more effectively.

Online urological educational material for medical students: can search engines be trusted?

OBJECTIVES

To determine the credibility of online urological information that medical students are likely to encounter, determine possible discrepancies between the credibility of information pertaining to different areas within urology (especially those less relevant to patients), and assess trends in the sponsorship of online urological educational material.

MATERIALS AND METHODS

Health on the Net (HON) principles were used as a validated benchmark to assess the reliability of websites that appeared in the first 150 results of a search using the Google search engine. A variety of urological search terms were used, grouped into three broad categories with varying relevance to patients and medical students. Further analysis focussed on the sponsorship of assessed websites.

RESULTS

A total of 5400 websites were assessed for validation over a set of 36 search terms. Only 843/5400 (15.6%) of these were HONcode accredited, indicating a large proportion of unverified and potentially unreliable information. Search engine rankings usually favoured accredited websites (P = 0.009), and accreditation peaked at 51.1% (184/360) in the first page of results, but sorting became weaker outside the highest search results. The percentage of accredited websites varied significantly between different subcategories of search terms such as conditions (18.3% [329/1800], P = 0.003) and procedures (13.5% [243/1800], P = 0.043). Governmental/educational and commercial sources supported the majority of websites assessed for sponsorship (21% [31/150] and 33% [49/150], respectively), and the former were more likely to rank highly in search results.

CONCLUSION

Online urological information frequently lacks validation and is often of indeterminate credibility. There is a marked decrease in the proportion of accredited websites beyond the highest-ranked results and when considering search categories more relevant to students and less relevant to patients. Students cannot necessarily rely on free online sources for accurate information and could benefit from the development of more rigorous novel tools and platforms.

Online information on face masks: analysis of websites in Italian and English returned by different search engines

OBJECTIVE

Countries have major differences in the acceptance of face mask use for the prevention of COVID-19. This work aims at studying the information online in different countries in terms of information quality and content.

DESIGN

Content analysis.

METHOD

We analysed 450 webpages returned by searching the string 'are face masks dangerous' in Italy, the UK and the USA using three search engines (Bing, Duckduckgo and Google) in August 2020. The type of website and the stance about masks were assessed by two raters for each language and inter-rater agreement reported as Cohen's kappa. The text of the webpages was collected from the web using WebBootCaT and analysed using a corpus analysis software to identify issues mentioned.

RESULTS

Most pages were news outlets, and few (2%-6%) from public health agencies. Webpages with a negative stance on masks were more frequent in Italian (28%) than English (19%). Google returned the highest number of mask-positive pages and Duckduckgo the lowest. Google also returned the lowest number of pages mentioning conspiracy theories and Duckduckgo the highest. Webpages in Italian scored lower than those in English in transparency (reporting authors, their credentials and backing the information with references). When issues about the use of face masks were analysed, mask effectiveness was the most discussed followed by hypercapnia (accumulation of carbon dioxide), contraindication in respiratory disease and hypoxia, with issues related to their contraindications in mental health conditions and disability mentioned by very few pages.

CONCLUSIONS

This study suggests that: (1) public health agencies should increase their web presence in providing correct information on face masks; (2) search engines should improve the information quality criteria in their ranking; (3) the public should be more informed on issues related to the use of masks and disabilities, mental health and stigma arising for those people who cannot wear masks.

Quality and readability of web-based Arabic health information on periodontal disease

BACKGROUND

Currently, the Internet seems to be a helpful tool for obtaining information about everything that we think about, including diseases, their prevention and treatment approaches. However, doubts exist regarding the quality and readability of such information. This study sought to assess the quality and readability of web-based Arabic information on periodontal disease.

METHODS

In this infodemiological study, the Google, Yahoo!, and Bing search engines were searched using specific Arabic terms on periodontal disease. The first 100 consecutive websites from each engine were obtained. The eligible websites were categorized as commercial, health/professional, journalism, and other. The following tools were applied to assess the quality of the information on the included websites: the Health on the Net Foundation Code of Conduct (HONcode), the Journal of the American Medical Association (JAMA) benchmarks, and the DISCERN tool. The readability was assessed using an online readability tool.

RESULTS

Of the 300 websites, 89 were eligible for quality and readability analyses. Only two websites (2.3%) were HONcode certified. Based on the DISCERN tool, 43 (48.3%) websites had low scores. The mean score of the JAMA benchmarks was 1.6 ± 1.0, but only 3 (3.4%) websites achieved "yes" responses for all four JAMA criteria. Based on the DISCERN tool, health/professional websites revealed the highest quality of information compared to other website categories. Most of the health/professional websites revealed moderate-quality information, while 55% of the commercial websites, 66% of journalism websites, and 43% of other websites showed poor quality information. Regarding readability, most of the analyzed websites presented simple and readable written content.

CONCLUSIONS

Aside from readable content, Arabic health information on the analyzed websites on periodontal disease is below the required level of quality.

Automatic Identification of Information Quality Metrics in Health News Stories

Objective: Many online and printed media publish health news of questionable trustworthiness and it may be difficult for laypersons to determine the information quality of such articles. The purpose of this work was to propose a methodology for the automatic assessment of the quality of health-related news stories using natural language processing and machine learning.

Materials and Methods: We used a database from the website HealthNewsReview.org that aims to improve the public dialogue about health care. HealthNewsReview.org developed a set of criteria to critically analyze health care interventions' claims. In this work, we attempt to automate the evaluation process by identifying the indicators of those criteria using natural language processing-based machine learning on a corpus of more than 1,300 news stories. We explored features ranging from simple n-grams to more advanced linguistic features and optimized the feature selection for each task. Additionally, we experimented with the use of pre-trained natural language model BERT.

Results: For some criteria, such as mention of costs, benefits, harms, and “disease-mongering,” the evaluation results were promising with an F₁ measure reaching 81.94%, while for others the results were less satisfactory due to the dataset size, the need of external knowledge, or the subjectivity in the evaluation process.

Conclusion: These used criteria are more challenging than those addressed by previous work, and our aim was to investigate how much more difficult the machine learning task was, and how and why it varied between criteria. For some criteria, the obtained results were promising; however, automated evaluation of the other criteria may not yet replace the manual evaluation process where human experts interpret text senses and make use of external knowledge in their assessment.

Evaluation of information quality on the internet for periodontal disease patients

OBJECTIVE

To evaluate the quality of accessible information on periodontal diseases on the Internet using different scales.

MATERIALS AND METHODS

A search was performed using the Google search engine with questions about periodontal disease symptoms. The first 30 web sites obtained after searching for each question were evaluated. Duplicate web sites, advertisements, discussion groups, links to research articles, videos, and images were excluded. A total of 90 web sites were included and evaluated with Health on the Net Code of Conduct Certification (HONCode) presence, Journal of American Medical Association (JAMA) benchmarks, and the quality criteria for consumer health information (DISCERN) toolkit.

RESULTS

Only 27.8% of the analyzed web sites contain HONCode certificates. No webpages fulfilled all JAMA criteria, whereas 32.2% of the web sites did not provide any of them. Majority of the web sites' (44.4%) overall rating score was 2 with the DISCERN instrument. In the DISCERN Section Scores comparison between various types of web sites, information web site scores were higher than Dental Health Center Web sites in Section 1 and Section 3 scores (p = .000 and p = .001, respectively).

CONCLUSION

Overall quality of periodontal information based on patients' questions on the Internet has serious shortcomings especially in terms of attribution and the quality of information on treatment choices.

Trend of influenza vaccine Facebook posts in last 4 years: a content analysis

BACKGROUND

Decreasing immunization rates may be partly due to antivaccine campaigns and other sources of misinformation available on social media, particularly on Facebook. Given the potential impact of this medium for communicating vaccine-related information, it is important to analyze the trend of information available on Facebook.

METHODS

We searched Facebook on August 15, 2018 to obtain posts containing relevant health information on influenza vaccine in years 2015-2018. We collected information regarding nature of the post (eg, pro-, antivaccine, and informational), number of shares and likes received, and ease of reading for each post. We evaluated these characteristics by year and type of post in our exploratory analyses.

RESULTS

The proportion of pro-vaccine posts has increased compared to antivaccine and informational posts since 2016. There was no correlation between ease of reading and popularity of posts. Although the language of antivaccine posts was complex, they were shared and liked more than pro-vaccine posts. The pro-vaccine personal post by a nurse was the most popular in our study (shared over 46,000 times) in 2018.

CONCLUSIONS

Though the number of pro-vaccine posts increased, antivaccine posts remained more popular. The government agency may use an emotive personal family-oriented message to promote vaccination.

Quality and readability of online information on ankylosing spondylitis

INTRODUCTION/OBJECTIVE

Obtaining online health-related information is becoming increasingly popular among patients. The attainment of information through websites is easy and practical, but there is no mechanism to check the accuracy and quality of this information. This leads to concerns about information from websites. Therefore, we aimed to evaluate the quality and readability of ankylosing spondylitis-related websites in this study.

METHODS

This is a descriptive study. Websites were searched on a popular search engine with the search term ankylosing spondylitis on March 2, 2019. We recorded the URLs of the first 200 websites listed in the query results. Typologies, quality, and readability were evaluated on these websites. Websites were divided into eight categories (commercial, government, health portal, news, non-profit, professional, scientific journal, and others) according to typology. The JAMA scoring system and the presence of HONcode certification were used to assess the quality. The Flesch-Kincaid grade and the Simple Measure of Gobbledygook were used to evaluate the readability.

RESULTS

Of the websites analyzed, 46% were in the high-quality group. We found that scientific journals and news were of higher quality, and commercial and other websites were of poorer quality. The average readability grades of the websites were 8.59 ± 2.42 and 7.33 ± 1.54, which were slightly worse than the recommended value. Additionally, the readability grades were significantly higher on high-quality websites (p < 0.05).

CONCLUSION

The quality of information on websites is variable. High-quality information about ankylosing spondylitis is available online, particularly from scientific journals and news. The poor readability of websites that provide high-quality information is a problem for patients with low health literacy. Editors should take into account readability while aiming to present high-quality information on websites. Key Points • Websites have become an important source of health-related information in parallel with the increase in internet use. • Less than half of the ankylosing spondylitis-related websites (46%) were of high quality according to JAMA scores. • The average readability grades of the ankylosing spondylitis-related websites were slightly worse than the recommended values. • High-quality websites had higher readability grades. Therefore, high-quality websites may not be understood by patients with low literacy levels. • No significant difference was found between the websites on the first page (n = 10) and remaining websites (n = 102) in terms of quality and readability.

Influenza vaccination discourse in major Canadian news media, 2017-2018

Influenza vaccine uptake is less-than-ideal in many jurisdictions, including Canada. In this study we sought to assess news articles relating to influenza vaccination by major Canadian newspapers during a six-month period relatively congruent to the seasonal influenza outbreak for 2017-2018. We identified 116 unique articles published between August 16, 2017 and February 15, 2018, then developed and applied a coding frame to them. Influenza vaccination was portrayed primarily positively (74.14%), sometimes negatively (14.66%), and occasionally neutrally (11.21%). Articles were most commonly focused on news about the prevalence, or amount of harm/death caused by, the influenza virus (31.03%), or on public announcements primarily concerning influenza vaccination (17.24%). Benefits of influenza vaccination were often stated (59.48%), most commonly including reduction in disease (47.41%) and protection of vulnerable individuals (26.72%). Issues or problems with influenza vaccination were also often stated (55.17%), most commonly relating to low or non-effectiveness of the vaccine (43.10%). Most articles stated that people should get vaccinated (65.52%). Canadian newspaper articles generally support the scientific consensus that influenza vaccination is a highly positive intervention. Nonetheless, a clear picture of the true value of influenza vaccination may sometimes be missing in articles focusing on low effectiveness and lacking any mention of vaccination's positive value. Overall, we can reasonably conclude that, in Canada, misinformation and antivaccination rhetoric are coming primarily from sources other than newspapers.

Fake News or Weak Science? Visibility and Characterization of Antivaccine Webpages Returned by Google in Different Languages and Countries

The 1998 Lancet paper by Wakefield et al., despite subsequent retraction and evidence indicating no causal link between vaccinations and autism, triggered significant parental concern. The aim of this study was to analyze the online information available on this topic. Using localized versions of Google, we searched “autism vaccine” in English, French, Italian, Portuguese, Mandarin, and Arabic and analyzed 200 websites for each search engine result page (SERP). A common feature was the newsworthiness of the topic, with news outlets representing 25–50% of the SERP, followed by unaffiliated websites (blogs, social media) that represented 27–41% and included most of the vaccine-negative websites. Between 12 and 24% of websites had a negative stance on vaccines, while most websites were pro-vaccine (43–70%). However, their ranking by Google varied. While in Google.com, the first vaccine-negative website was the 43rd in the SERP, there was one vaccine-negative webpage in the top 10 websites in both the British and Australian localized versions and in French and two in Italian, Portuguese, and Mandarin, suggesting that the information quality algorithm used by Google may work better in English. Many webpages mentioned celebrities in the context of the link between vaccines and autism, with Donald Trump most frequently. Few websites (1–5%) promoted complementary and alternative medicine (CAM) but 50–100% of these were also vaccine-negative suggesting that CAM users are more exposed to vaccine-negative information. This analysis highlights the need for monitoring the web for information impacting on vaccine uptake.

The involvement of Canadian physicians in promoting and providing unproven and unapproved stem cell interventions

Background

Direct to consumer offerings of unproven stem cell interventions (SCIs) is a pressing scientific and policy issue. According to media reports, providers of SCIs have emerged in Canada. This study provides the first systematic scan of Canadian providers and associated trends and claims.

Methods

The study sample consisted of 15 websites retrieved from a Google™ keyword search. The websites were assessed by a rater using a peer-reviewed coding frame that queried treatment location, stem cell offerings, treatment claims, supporting evidence, and legal and regulatory compliance. A second rater reviewed a subset of the websites for purposes of inter-rater reliability. Disagreements between raters were resolved by consensus. Data collected by the raters was analyzed in SPSS.

Results

Physicians are the dominant treatment providers in Canada. Providers operate in urban and semi-urban areas in the most populous provinces. SCIs provided are mainly autologous adult stem cells for multiple conditions including musculoskeletal disorders, spinal cord injury (SCI) and diabetes. Efficacy and benefits of treatment are prominently and positively portrayed, while risks are not mentioned or portrayed as trivial. Regulatory concerns are not discussed.

Conclusions

The involvement of physicians in promoting and providing unproven and unapproved SCIs raises significant ethical, legal and regulatory concerns. Treatment claims and trends appear to contravene applicable professional standards, statutory obligations, and consumer protection laws. While the number of providers observed is still marginal, urgent and proactive regulatory response is needed to prevent proliferation of a potentially exploitative and harmful market for unproven SCIs in Canada.

Electronic supplementary material

The online version of this article (10.1186/s12910-018-0273-6) contains supplementary material, which is available to authorized users.

A comprehensive analysis of Italian web pages mentioning squalene-based influenza vaccine adjuvants reveals a high prevalence of misinformation

Squalene-based adjuvants have been included in influenza vaccines since 1997. Despite several advantages of adjuvanted seasonal and pandemic influenza vaccines, laypeople's perception of such formulations may be hesitant or even negative under certain circumstances. Moreover, in Italian, the term “squalene” has the same root as such common words as “shark” (squalo), “squalid” and “squalidness” that tend to have negative connotations. This study aimed to quantitatively and qualitatively analyze a representative sample of Italian web pages mentioning squalene-based adjuvants used in influenza vaccines. Every effort was made to limit the subjectivity of judgments. Eighty-four unique web pages were assessed. A high prevalence (47.6%) of pages with negative or ambiguous attitudes toward squalene-based adjuvants was established. Compared with web pages reporting balanced information on squalene-based adjuvants, those categorized as negative/ambiguous had significantly lower odds of belonging to a professional institution [adjusted odds ratio (aOR) = 0.12, p = .004], and significantly higher odds of containing pictures (aOR = 1.91, p = .034) and being more readable (aOR = 1.34, p = .006). Some differences in wording between positive/neutral and negative/ambiguous web pages were also observed. The most common scientifically unsound claims concerned safety issues and, in particular, claims linking squalene-based adjuvants to the Gulf War Syndrome and autoimmune disorders. Italian users searching the web for information on vaccine adjuvants have a high likelihood of finding unbalanced and misleading material. Information provided by institutional websites should be not only evidence-based but also carefully targeted towards laypeople. Conversely, authors writing for non-institutional websites should avoid sensationalism and provide their readers with more balanced information.

Online Information on Antioxidants: Information Quality Indicators, Commercial Interests, and Ranking by Google

The idea that antioxidant supplements can prevent or cure many diseases is extremely popular. To study the public understanding of antioxidants on the Web, we searched the term “antioxidants” in http://Google.com and analyzed 200 websites in terms of typology (news, commercial, professional, health portal, no-profit or government organization, scientific journals), disease or biological process mentioned (aging, immunity, neurological disease, diabetes, arthritis, etc.), and stance toward antioxidants, whether neutral, positive, or negative. Commercial and news websites were prevalent (over half of the total) but not in the top 10 returned by Google, where the most frequent were health portals, government, and professional websites. Among the diseases mentioned, cancer was the first, followed by vascular and eye diseases. A negative stance toward supplements was prevalent in the whole search, and this was even more evident for cancer. Information on aging or immunity had the largest proportion of pro-supplement and commercial websites. This study shows that some diseases are highly associated with antioxidants on the Internet and that information on antioxidants in aging and immunity is more likely to describe the positive effects of antioxidant supplements.

The importance of the patient voice in vaccination and vaccine safety-are we listening?

Much has been written about the patient-physician relationship over the years. This relationship is essential in maintaining trust in the complex arena of modern diagnostic techniques, treatment and prevention, including vaccines and vaccine safety. However, a great deal of this material was written from the viewpoint of clinicians and academics. The patient voice may be positive or negative, fragmented or complex. Information sources are weighed and treated differently, according to the value system and risk perceptions of the individual. In post-trust societies, when people have less confidence in health authorities, communication needs to be more than a paternalistic top-down process. Notions of empowerment and individual patient choice are becoming crucial in medical care. The 'voice of the patient', which includes healthy individuals receiving vaccines, needs to be heard, considered and addressed. With respect to childhood immunizations, this will be the voice of the parent or caregiver. The key to addressing any concerns could be to listen more and to develop a communication style that is trust-based and science-informed. Regulatory agencies are encouraging clinical and patient-reported outcomes research under the umbrella of personalized medicine, and this is an important step forward. This paper attempts to reflect the paradigm shift towards increasing attention to the patient voice in vaccination and vaccine safety.