Librarian and Researcher Assessments of Search Result Relevance: How Well Do They Align?

Conducting comprehensive but efficient literature searches for complex evidence syntheses involves selecting databases that will retrieve the greatest number of relevant results on the question. Lack of a comprehensive single database on allied health educational topics challenges those seeking such literature. In this study, six participants contributed research questions on instructional methods and materials for allied health patients, caregivers, and future health professionals. Two health sciences librarians created search strategies for these questions and searched eleven databases. Both the librarians and the six participants evaluated the search results using a rubric based on PICO to assess extent of alignment between the librarians' and requestors' relevance judgments. Intervention, Outcome, and Assessment Method constituted the most frequent bases for assessments of relevance by both librarians and participants. The librarians were more restrictive in all of their assessments except in a preliminary search yielding twelve citations without abstracts. The study's results could be used to identify effective techniques for reference interviewing, selecting databases, and weeding search results.

The Criteria People Use in Relevance Decisions on Health Information: An Analysis of User Eye Movements When Browsing a Health Discussion Forum

Background

People are increasingly accessing health-related social media sites, such as health discussion forums, to post and read user-generated health information. It is important to know what criteria people use when deciding the relevance of information found on health social media websites, in different situations.

Objective

The study attempted to identify the relevance criteria that people use when browsing a health discussion forum, in 3 types of use contexts: when seeking information for their own health issue, when seeking for other people’s health issue, and when browsing without a particular health issue in mind.

Methods

A total of 58 study participants were self-assigned to 1 of the 3 use contexts or information needs and were asked to browse a health discussion forum, HealthBoards.com. In the analysis, browsing a discussion forum was divided into 2 stages: scanning a set of post surrogates (mainly post titles) in the summary result screen and reading a detailed post content (including comments by other users). An eye tracker system was used to capture participants’ eye movement behavior and the text they skim over and focus (ie, fixate) on during browsing. By analyzing the text that people’s eyes fixated on, the types of health information used in the relevance judgment were determined. Post-experiment interviews elicited participants’ comments on the relevance of the information and criteria used.

Results

It was found that participants seeking health information for their own health issue focused significantly more on the poster’s symptoms, personal history of the disease, and description of the disease (P=.01, .001, and .02). Participants seeking for other people’s health issue focused significantly more on cause of disease, disease terminology, and description of treatments and procedures (P=.01, .01, and .02). In contrast, participants browsing with no particular issue in mind focused significantly more on general health topics, hot topics, and rare health issues (P=.01, .01, and .01).

Conclusion

Users browsing for their own health issues used mainly case-based relevance criteria to relate the poster's health situation to their own. Participants seeking for others’ issues used mostly general knowledge–based criteria, whereas users with no particular issue in mind used general interest– and curiosity-based criteria.